Jul 29 22:44:22.150676 ip-10-0-131-193 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 22:44:22.150687 ip-10-0-131-193 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 22:44:22.150697 ip-10-0-131-193 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 22:44:22.151003 ip-10-0-131-193 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 22:44:32.303076 ip-10-0-131-193 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 22:44:32.303092 ip-10-0-131-193 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 25f140406e314008b7be9c2ad909d43e --
Jul 29 22:46:22.656733 ip-10-0-131-193 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 22:46:23.014947 ip-10-0-131-193 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:23.014947 ip-10-0-131-193 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 22:46:23.014947 ip-10-0-131-193 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:23.014947 ip-10-0-131-193 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 22:46:23.014947 ip-10-0-131-193 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 22:46:23.016705 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.016594    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 22:46:23.021090 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021068    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:23.021090 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021087    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:23.021090 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021092    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:23.021090 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021096    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021104    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021110    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021115    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021120    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021124    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021128    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021132    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021136    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021140    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021144    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021148    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021152    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021156    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021161    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021165    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021168    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021170    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021173    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:23.021259 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021176    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021179    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021181    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021184    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021187    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021190    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021192    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021195    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021198    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021200    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021203    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021205    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021208    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021211    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021213    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021217    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021220    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021223    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021226    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021229    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:23.021752 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021231    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021235    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021238    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021240    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021244    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021247    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021249    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021251    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021255    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021257    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021260    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021263    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021265    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021269    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021273    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021276    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021278    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021281    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021283    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021285    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:23.022242 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021288    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021291    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021293    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021295    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021298    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021301    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021303    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021307    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021310    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021312    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021315    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021318    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021321    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021323    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021326    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021328    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021331    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021334    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021336    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021339    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:23.022863 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021341    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021344    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021348    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021351    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021353    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021356    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021358    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021360    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021363    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021365    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021368    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021371    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021390    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021393    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021397    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021400    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.021403    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022106    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022113    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022116    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:23.023361 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022119    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022122    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022124    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022127    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022130    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022132    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022135    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022138    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022140    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022143    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022145    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022148    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022151    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022153    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022157    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022159    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022162    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022166    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022168    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:23.023872 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022171    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022173    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022176    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022179    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022181    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022184    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022186    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022191    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022194    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022197    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022200    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022203    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022206    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022209    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022212    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022214    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022217    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022220    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022223    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:23.024349 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022225    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022228    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022231    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022234    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022236    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022239    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022242    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022245    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022248    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022252    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022256    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022258    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022261    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022264    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022267    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022269    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022272    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022275    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022277    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022280    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:23.024844 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022282    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022285    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022287    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022290    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022292    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022296    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022299    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022302    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022305    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022307    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022310    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022312    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022315    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022317    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022319    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022322    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022324    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022327    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022330    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:23.025355 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022333    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022336    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022340    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022345    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022348    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022351    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022354    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022356    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022359    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022362    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022364    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022367    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022369    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022372    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022388    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022391    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022394    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022397    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022400    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022833    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:23.025840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022839    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022842    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022845    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022848    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022851    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022854    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022856    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022859    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022862    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022864    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022867    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022870    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022872    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022875    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022880    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022885    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022892    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022895    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022898    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:23.026337 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022902    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022905    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022908    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022911    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022913    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022916    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022919    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022922    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022924    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022926    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022929    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022932    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022935    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022938    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022940    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022943    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022946    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022948    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022951    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:23.026840 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022953    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022957    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022960    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022962    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022965    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022967    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022970    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022972    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022975    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022978    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022982    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022985    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022987    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022990    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022993    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022995    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.022998    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023000    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023003    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:23.027307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023005    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023008    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023010    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023013    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023015    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023018    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023021    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023023    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023026    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023028    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023031    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023033    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023036    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023038    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023041    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023043    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023046    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023049    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023051    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023054    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:23.027783 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023056    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023059    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023062    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023066    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023069    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023071    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023076    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023079    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023082    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023085    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023088    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023090    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023093    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023096    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023098    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023101    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023103    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023109    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023112    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023115    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:23.028307 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023117    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023192    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023202    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023211    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023217    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023223    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023228    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023235    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023240    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023243    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023246    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023250    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023253    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023256    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023259    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023263    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023267    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023270    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023273    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023279    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023281    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023285    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023288    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023291    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 22:46:23.028815 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023295    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023298    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023301    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023304    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023307    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023311    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023314    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023317    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023320    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023324    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023327    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023330    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023333    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023336    2580 flags.go:64] FLAG: --enable-server="true"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023339    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023343    2580 flags.go:64] FLAG: --event-burst="100"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023346    2580 flags.go:64] FLAG: --event-qps="50"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023349    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023352    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023356    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023362    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023367    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023385    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023391    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023394    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 22:46:23.029404 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023397    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023401    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023404    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023407    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023410    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023412    2580 flags.go:64] FLAG: --feature-gates=""
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023416    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023419    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023422    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023425    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023428    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023431    2580 flags.go:64] FLAG: --help="false"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023435    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-193.ec2.internal"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023438    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023441    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023444    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023447    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023451    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023454    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023456    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023459    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023462    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023465    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023468    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 22:46:23.030115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023471    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023474    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023477    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023480    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023483    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023485    2580 flags.go:64] FLAG: --lock-file=""
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023490    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023492    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023496    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023501    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023504    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023507    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023511    2580 flags.go:64] FLAG: --logging-format="text"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023513    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023517    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023519    2580 flags.go:64] FLAG: --manifest-url=""
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023522    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023527    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023530    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023534    2580 flags.go:64] FLAG: --max-pods="110"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023538    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023541    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023544    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023547    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023550    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 22:46:23.030731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023553    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023556    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023563    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023566    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023569    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023572    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023575    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023581    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023584    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023587    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023590    2580 flags.go:64] FLAG: --port="10250"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023593    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023596    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ca6d1813fcc8cad1"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023600    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023604    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023607    2580 flags.go:64] FLAG: --register-node="true"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023610    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023614    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023617    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023620    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023623    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023626    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023630    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023633    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023635    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 22:46:23.031345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023638    2580 flags.go:64] FLAG: --runonce="false"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023641    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023645    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023648    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023651    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023654    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023657    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023660    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023663    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023666    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023669    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023672    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023674    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023678    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023680    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023683    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023688    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023691    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023694    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023698    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023701    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023704    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023707    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023710    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023713    2580 flags.go:64] FLAG: --v="2"
Jul 29 22:46:23.031988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023717    2580 flags.go:64] FLAG: --version="false"
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023721    2580 flags.go:64] FLAG: --vmodule=""
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023725    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.023728    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023841    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023844    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023847    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023850    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023853    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023857    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023860    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023863    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023865    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023868    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023870    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023873    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023876    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023878    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023881    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023884    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:23.032612 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023887    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023890    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023893    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023895    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023898    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023901    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023903    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023906    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023909    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023912    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023914    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023917    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023919    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023922    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023925    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023928    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023930    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023934    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023938    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023940    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:23.033108 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023943    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023947    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023949    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023952    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023954    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023957    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023960    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023963    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023965    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023968    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023970    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023973    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023975    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023978    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023980    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023983    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023985    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023988    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023991    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023993    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:23.033661 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023996    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.023998    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024001    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024003    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024006    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024009    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024011    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024014    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024017    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024019    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024022    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024025    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024027    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024031    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024033    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024036    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024039    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024043    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024046    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:23.034158 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024050    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024053    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024057    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024059    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024062    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024064    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024067    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024069    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024072    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024074    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024077    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024080    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024082    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024085    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024087    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024090    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024092    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024095    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024098    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024100    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:23.034641 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024103    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024106    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024108    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.024111    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.024117    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.031804    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.031820    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.031842    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.031849    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031904    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031910    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031913    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031917    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031919    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:23.035137 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031922    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031925    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031929    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031933    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031936    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031939    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031942    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031945    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031948    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031950    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031953    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031956    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031958    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031961    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031964    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031966    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031969    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031971    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031974    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031976    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:23.035527 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031979    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031982    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031985    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031988    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031991    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031994    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.031997    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032001    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032005    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032008    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032010    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032013    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032016    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032018    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032021    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032023    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032026    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032029    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032031    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:23.036022 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032035    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032039    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032042    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032045    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032048    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032051    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032054    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032057    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032059    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032062    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032064    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032067    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032070    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032072    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032075    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032078    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032081    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032084    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032087    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032089    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:23.036669 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032092    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032095    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032097    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032099    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032102    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032105    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032107    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032110    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032112    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032115    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032118    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032120    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032123    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032125    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032128    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032131    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032133    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032136    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032138    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032140    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:23.037228 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032143    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032146    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032148    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032151    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032153    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032156    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032159    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032162    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032165    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032167    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032170    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032173    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032175    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032178    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032180    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:23.037744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.032185    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032297    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032304    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032308    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032311    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032314    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032316    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032319    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032322    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032324    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032327    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032329    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032332    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032334    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032337    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032339    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032342    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032344    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032347    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032349    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 22:46:23.038202 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032352    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032355    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032357    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032359    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032362    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032366    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032369    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032371    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032385    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032389    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032391    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032394    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032397    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032399    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032402    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032404    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032407    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032410    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032412    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032415    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 22:46:23.038779 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032418    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032420    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032423    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032425    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032428    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032430    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032433    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032435    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032438    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032440    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032443    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032445    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032448    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032450    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032453    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032455    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032458    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032461    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032464    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032466    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 22:46:23.039684 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032469    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032471    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032474    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032477    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032480    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032482    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032486    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032489    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032492    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032495    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032497    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032500    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032502    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032505    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032508    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032510    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032513    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032515    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032518    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 22:46:23.040426 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032520    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032524    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032526    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032529    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032531    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032534    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032536    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032539    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032541    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032544    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032547    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032549    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032552    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032554    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032557    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032560    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032563    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032565    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032569    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 22:46:23.040970 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032572    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:23.032575    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.032580    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.032693    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.035069    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.036286    2580 server.go:1019] "Starting client certificate rotation"
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.036403    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.036443    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.038711    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 22:46:23.041567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.039828    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 22:46:23.049728 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.049711    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 29 22:46:23.052617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.052604    2580 log.go:25] "Validated CRI v1 image API"
Jul 29 22:46:23.053405 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.053392    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 22:46:23.057789 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.057770    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90146a79-b312-4efc-9f37-e8bdb9dd98b4:/dev/nvme0n1p3 f92aab06-5c9e-4dea-9ed4-5aa82854374b:/dev/nvme0n1p4]
Jul 29 22:46:23.057856 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.057788    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 22:46:23.062743 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.062629    2580 manager.go:216] Machine: {Timestamp:2026-07-29 22:46:23.061457514 +0000 UTC m=+0.299558105 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088002 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec234d43d1870b2850ef7423029d8134 SystemUUID:ec234d43-d187-0b28-50ef-7423029d8134 BootID:25f14040-6e31-4008-b7be-9c2ad909d43e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:eb:41:24:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:eb:41:24:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:55:71:eb:21:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 22:46:23.062743 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.062738    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 22:46:23.062862 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.062815    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 22:46:23.065321 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.065291    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 22:46:23.065511 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.065325    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 22:46:23.066092 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.066079    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 22:46:23.066144 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.066099    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 22:46:23.066144 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.066110    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 22:46:23.066730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.066717    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 22:46:23.067736 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.067723    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 22:46:23.070236 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070219    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 22:46:23.070488 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070475    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 22:46:23.070659 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070648    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 22:46:23.070725 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070690    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 22:46:23.070725 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070705    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 22:46:23.070813 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070726    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 22:46:23.070813 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070740    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 29 22:46:23.070813 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.070765    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 22:46:23.071119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.071105    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 22:46:23.071182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.071128    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 22:46:23.073119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.073093    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 22:46:23.074836 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.074822    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 22:46:23.074898 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.074854    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 22:46:23.076049 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076036    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076057    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076068    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076076    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076085    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076091    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076100    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076105    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076112    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 22:46:23.076119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076118    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 22:46:23.076352 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076127    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 22:46:23.076352 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076132    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 22:46:23.076819 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076810    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 22:46:23.076849 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.076821    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 22:46:23.080145 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.080127    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 22:46:23.080281 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.080270    2580 server.go:1268] "Started kubelet"
Jul 29 22:46:23.080383 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.080346    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 22:46:23.080573 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.080554    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 22:46:23.080643 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.080589    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 22:46:23.080704 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.080621    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 22:46:23.080704 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.080621    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 22:46:23.081758 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.081742    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 22:46:23.082013 ip-10-0-131-193 systemd[1]: Started Kubernetes Kubelet.
Jul 29 22:46:23.086709 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.086683    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 22:46:23.087466 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.087448    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 22:46:23.090968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.090952    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 22:46:23.090968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.090962    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 22:46:23.091115 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.091021    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:23.091458 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091441    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 22:46:23.091547 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091478    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 22:46:23.091547 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091493    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 22:46:23.091654 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091612    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 22:46:23.091654 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091634    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 22:46:23.091967 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091947    2580 factory.go:153] Registering CRI-O factory
Jul 29 22:46:23.091967 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.091969    2580 factory.go:223] Registration of the crio container factory successfully
Jul 29 22:46:23.092124 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092016    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 22:46:23.092124 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092026    2580 factory.go:55] Registering systemd factory
Jul 29 22:46:23.092124 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092033    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 29 22:46:23.092124 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092054    2580 factory.go:103] Registering Raw factory
Jul 29 22:46:23.092124 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092066    2580 manager.go:1196] Started watching for new ooms in manager
Jul 29 22:46:23.093140 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092517    2580 manager.go:349] Starting recovery of all containers
Jul 29 22:46:23.093140 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.092594    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w5vjt"
Jul 29 22:46:23.093610 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.091942    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-193.ec2.internal.18c6e42cbbd92260  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-193.ec2.internal,UID:ip-10-0-131-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-193.ec2.internal,},FirstTimestamp:2026-07-29 22:46:23.080243808 +0000 UTC m=+0.318344445,LastTimestamp:2026-07-29 22:46:23.080243808 +0000 UTC m=+0.318344445,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-193.ec2.internal,}"
Jul 29 22:46:23.095778 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.095755    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 22:46:23.095924 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.095864    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 22:46:23.100121 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.100098    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w5vjt"
Jul 29 22:46:23.101880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.101848    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 22:46:23.102484 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.102471    2580 manager.go:354] Recovery completed
Jul 29 22:46:23.104052 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.104025    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 29 22:46:23.107561 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.107551    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:23.109305 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109290    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:23.109367 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109317    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:23.109367 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109328    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:23.109818 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109805    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 22:46:23.109818 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109817    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 22:46:23.109884 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109833    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 22:46:23.109884 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.109853    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:23.110522 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.110504    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:23.110559 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.110529    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:23.110559 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.110538    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:23.111525 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.111509    2580 policy_none.go:49] "None policy: Start"
Jul 29 22:46:23.111575 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.111530    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 22:46:23.111575 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.111542    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 22:46:23.111575 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.111494    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-193.ec2.internal.18c6e42cbd948f1f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-193.ec2.internal,UID:ip-10-0-131-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-193.ec2.internal,},FirstTimestamp:2026-07-29 22:46:23.109304095 +0000 UTC m=+0.347404687,LastTimestamp:2026-07-29 22:46:23.109304095 +0000 UTC m=+0.347404687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-193.ec2.internal,}"
Jul 29 22:46:23.112531 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.112520    2580 policy_none.go:47] "Start"
Jul 29 22:46:23.153229 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153084    2580 manager.go:337] "Starting Device Plugin manager"
Jul 29 22:46:23.153367 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.153244    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 22:46:23.153367 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153257    2580 server.go:87] "Starting device plugin registration server"
Jul 29 22:46:23.153512 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153502    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 22:46:23.153544 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153514    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 22:46:23.153652 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153632    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 22:46:23.153750 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153730    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 22:46:23.153750 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:23.153743    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 22:46:23.154277 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.154248    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 22:46:23.154387 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:23.154312    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:24.562999 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.562964    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 22:46:24.564063 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.563008    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 22:46:24.564063 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.563032    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 22:46:24.564063 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.563066    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 22:46:24.565818 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.565801    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:24.583834 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.583817    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:24.584654 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.584636    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:24.584722 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.584670    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:24.584722 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.584686    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:24.584722 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.584717    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.593640 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.593621    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.593704 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.593644    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-193.ec2.internal\": node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:24.611546 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.611490    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:24.663538 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.663501    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"]
Jul 29 22:46:24.663665 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.663575    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:24.665512 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.665490    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:24.665602 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.665530    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:24.665602 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.665543    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:24.666829 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.666816    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:24.666922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.666905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.666972 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.666944    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:24.667556 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.667525    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:24.667556 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.667537    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:24.667556 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.667555    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:24.667556 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.667561    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:24.667738 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.667566    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:24.667738 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.667572    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:24.668688 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.668675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.668732 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.668699    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 22:46:24.669309 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.669289    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 22:46:24.669407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.669320    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 22:46:24.669407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.669334    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeHasSufficientPID"
Jul 29 22:46:24.700515 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.700492    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-193.ec2.internal\" not found" node="ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.704961 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.704942    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-193.ec2.internal\" not found" node="ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.711975 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.711959    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:24.722638 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.722620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f65bcea48aa9eea90fe0167e5920afd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal\" (UID: \"f65bcea48aa9eea90fe0167e5920afd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.722725 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.722651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f65bcea48aa9eea90fe0167e5920afd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal\" (UID: \"f65bcea48aa9eea90fe0167e5920afd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.722725 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.722678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04c84e703d5815c2c5d803ec17e0971e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.ec2.internal\" (UID: \"04c84e703d5815c2c5d803ec17e0971e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.812676 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.812627    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:24.822982 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.822957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04c84e703d5815c2c5d803ec17e0971e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.ec2.internal\" (UID: \"04c84e703d5815c2c5d803ec17e0971e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.823052 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.822987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f65bcea48aa9eea90fe0167e5920afd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal\" (UID: \"f65bcea48aa9eea90fe0167e5920afd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.823052 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.823004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f65bcea48aa9eea90fe0167e5920afd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal\" (UID: \"f65bcea48aa9eea90fe0167e5920afd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.823052 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.823045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f65bcea48aa9eea90fe0167e5920afd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal\" (UID: \"f65bcea48aa9eea90fe0167e5920afd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.823148 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.823071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/04c84e703d5815c2c5d803ec17e0971e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-193.ec2.internal\" (UID: \"04c84e703d5815c2c5d803ec17e0971e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.823148 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:24.823080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f65bcea48aa9eea90fe0167e5920afd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal\" (UID: \"f65bcea48aa9eea90fe0167e5920afd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:24.913450 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:24.913374    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.002995 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.002963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:25.009068 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.009045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:25.014256 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:25.014240    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.114774 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:25.114738    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.215275 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:25.215211    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.315714 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:25.315679    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.366226 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.366205    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 22:46:25.366605 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.366348    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 22:46:25.416185 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:25.416156    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.434923 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.434878    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 22:41:24 +0000 UTC" deadline="2028-02-05 15:54:44.859888868 +0000 UTC"
Jul 29 22:46:25.434923 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.434917    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13337h8m19.424975251s"
Jul 29 22:46:25.514171 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.514100    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:25.516795 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:25.516768    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-193.ec2.internal\" not found"
Jul 29 22:46:25.571598 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.571573    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:25.621204 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.621177    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:25.632297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.632276    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 22:46:25.633259 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.633241    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal"
Jul 29 22:46:25.641864 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.641848    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 22:46:25.655416 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:25.655391    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c84e703d5815c2c5d803ec17e0971e.slice/crio-7fb2f202c2b0f9cc517f08bad27495fbfde3109c9d56e6f27b591637ee2d4ef6 WatchSource:0}: Error finding container 7fb2f202c2b0f9cc517f08bad27495fbfde3109c9d56e6f27b591637ee2d4ef6: Status 404 returned error can't find the container with id 7fb2f202c2b0f9cc517f08bad27495fbfde3109c9d56e6f27b591637ee2d4ef6
Jul 29 22:46:25.655698 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:25.655679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65bcea48aa9eea90fe0167e5920afd3.slice/crio-22e36da667972a21566c3e497655cf2b5af1b316e7b821ecaef6bf8af8c24e81 WatchSource:0}: Error finding container 22e36da667972a21566c3e497655cf2b5af1b316e7b821ecaef6bf8af8c24e81: Status 404 returned error can't find the container with id 22e36da667972a21566c3e497655cf2b5af1b316e7b821ecaef6bf8af8c24e81
Jul 29 22:46:25.661332 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.661319    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:46:25.705114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:25.705085    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:26.401145 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.401098    2580 apiserver.go:52] "Watching apiserver"
Jul 29 22:46:26.410205 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.410179    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 22:46:26.410708 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.410681    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ts59j","openshift-multus/multus-p94q8","openshift-network-diagnostics/network-check-target-2psct","openshift-network-operator/iptables-alerter-cqgbv","kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f","openshift-image-registry/node-ca-hg8hl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal","openshift-multus/multus-additional-cni-plugins-bgxrn","openshift-multus/network-metrics-daemon-vct9g","openshift-ovn-kubernetes/ovnkube-node-grsjn","kube-system/global-pull-secret-syncer-x8j5c","kube-system/konnectivity-agent-c6vx2","openshift-cluster-node-tuning-operator/tuned-gcmzb"]
Jul 29 22:46:26.412729 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.412599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.414663 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.414634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.414925 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.414771    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.415185 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.415166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 22:46:26.415278 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.415197    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t55jd\""
Jul 29 22:46:26.415278 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.415249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.415943 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.415871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:26.415943 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.415934    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:26.416706 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.416689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m64b9\""
Jul 29 22:46:26.416976 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.416956    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.417205 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.417186    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.417801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.417365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 22:46:26.417801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.417484    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 22:46:26.417801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.417512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 22:46:26.417801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.417714    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 22:46:26.417801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.417756    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.418619 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.418600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.419340 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.419315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.419428 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.419353    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 22:46:26.419703 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.419682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cwhnn\""
Jul 29 22:46:26.420016 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.419997    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.420016 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.420010    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.420495 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.420478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.420609 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.420593    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.420887 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.420869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qz2v5\""
Jul 29 22:46:26.420977 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.420898    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 22:46:26.421911 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.421887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.421985 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.421916    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.422364 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.422124    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.422469 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.422429    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xltbk\""
Jul 29 22:46:26.423051 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.422908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 22:46:26.423492 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.423476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.423606 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.423555    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jhgqg\""
Jul 29 22:46:26.423969 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.423927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.424532 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.424274    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 22:46:26.424532 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.424397    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 22:46:26.424725 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.424707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.425067 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.425050    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 22:46:26.425143 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.425065    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 22:46:26.425355 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.425332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 22:46:26.425464 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.425449    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vq7gx\""
Jul 29 22:46:26.426566 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.426547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.428479 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.428461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:26.428569 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.428504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 22:46:26.428569 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.428536    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:26.428768 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.428747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7fwlw\""
Jul 29 22:46:26.430914 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.430894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-kubernetes\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.430998 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.430928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55a8937d-e59e-4e64-9f56-4fe485e115f1-iptables-alerter-script\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.430998 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.430954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-socket-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.431092 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431024    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-cnibin\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.431092 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-log-socket\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431092 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-var-lib-kubelet\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.431220 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-ovn\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431220 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431134    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-cni-bin\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431220 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysctl-conf\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.431220 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fr6\" (UniqueName: \"kubernetes.io/projected/e01dc35e-2818-4861-913c-f4df8df8f334-kube-api-access-69fr6\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-kubelet\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-run-netns\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431289    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-var-lib-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjrp\" (UniqueName: \"kubernetes.io/projected/ba52014f-ffe4-4963-b240-5d8271d599a0-kube-api-access-pwjrp\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:26.431408 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/062d05c7-b02b-464f-abb6-3c6f113ee0ce-serviceca\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ftrt\" (UniqueName: \"kubernetes.io/projected/062d05c7-b02b-464f-abb6-3c6f113ee0ce-kube-api-access-2ftrt\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-systemd-units\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysconfig\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-host\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-registration-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-slash\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-etc-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-modprobe-d\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-systemd\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-systemd\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-cni-netd\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-node-log\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.431730 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba52014f-ffe4-4963-b240-5d8271d599a0-ovn-node-metrics-cert\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-lib-modules\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysctl-d\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-sys\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqp26\" (UniqueName: \"kubernetes.io/projected/cc89222e-825d-469d-96f7-9ef4ee6c15a6-kube-api-access-bqp26\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-ovnkube-config\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-ovnkube-script-lib\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.431979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5efe24fd-9205-4070-805d-38ce688be1d8-etc-tuned\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvtl\" (UniqueName: \"kubernetes.io/projected/5efe24fd-9205-4070-805d-38ce688be1d8-kube-api-access-7nvtl\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432020    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/062d05c7-b02b-464f-abb6-3c6f113ee0ce-host\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5efe24fd-9205-4070-805d-38ce688be1d8-tmp\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.432448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-run\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55a8937d-e59e-4e64-9f56-4fe485e115f1-host-slash\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4hw\" (UniqueName: \"kubernetes.io/projected/55a8937d-e59e-4e64-9f56-4fe485e115f1-kube-api-access-6c4hw\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-env-overrides\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-device-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-os-release\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-sys-fs\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-system-cni-dir\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.432747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.433182 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.432834    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:26.435164 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.434013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 22:46:26.435164 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.434611    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 22:46:26.435164 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.434853    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5l9vz\""
Jul 29 22:46:26.451748 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.451729    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bttx8"
Jul 29 22:46:26.459163 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.459053    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bttx8"
Jul 29 22:46:26.522066 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.522033    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 22:46:26.532977 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.532909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-node-log\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.532977 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.532948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba52014f-ffe4-4963-b240-5d8271d599a0-ovn-node-metrics-cert\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.533195 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-lib-modules\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.533195 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-lib-modules\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.533319 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-node-log\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.533319 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.533319 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twc5d\" (UniqueName: \"kubernetes.io/projected/af595ab0-8515-4923-af82-6cdcd20b5c57-kube-api-access-twc5d\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:26.533319 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbbk\" (UniqueName: \"kubernetes.io/projected/df0d9576-7662-40f9-b242-50eb3acada25-kube-api-access-mdbbk\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.533498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533339    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysctl-d\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.533498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-sys\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.533498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.533498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533413    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 22:46:26.533498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-socket-dir-parent\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.533498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-cni-multus\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-conf-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqp26\" (UniqueName: \"kubernetes.io/projected/cc89222e-825d-469d-96f7-9ef4ee6c15a6-kube-api-access-bqp26\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-ovnkube-config\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533590    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-ovnkube-script-lib\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5efe24fd-9205-4070-805d-38ce688be1d8-etc-tuned\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvtl\" (UniqueName: \"kubernetes.io/projected/5efe24fd-9205-4070-805d-38ce688be1d8-kube-api-access-7nvtl\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-os-release\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/062d05c7-b02b-464f-abb6-3c6f113ee0ce-host\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.533760 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5efe24fd-9205-4070-805d-38ce688be1d8-tmp\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-cni-binary-copy\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-daemon-config\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0d9576-7662-40f9-b242-50eb3acada25-tmp-dir\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysctl-d\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-sys\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-hostroot\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-run\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.533980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55a8937d-e59e-4e64-9f56-4fe485e115f1-host-slash\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534005    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df0d9576-7662-40f9-b242-50eb3acada25-hosts-file\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4hw\" (UniqueName: \"kubernetes.io/projected/55a8937d-e59e-4e64-9f56-4fe485e115f1-kube-api-access-6c4hw\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-etc-kubernetes\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-env-overrides\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-cni-binary-copy\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.534242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-device-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-os-release\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-k8s-cni-cncf-io\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-cni-bin\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13db78f9-f4eb-405b-b632-2a9eed835018-kubelet-config\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-os-release\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/062d05c7-b02b-464f-abb6-3c6f113ee0ce-host\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-sys-fs\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534419    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-sys-fs\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-system-cni-dir\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-run\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-device-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-env-overrides\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55a8937d-e59e-4e64-9f56-4fe485e115f1-host-slash\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.535069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-system-cni-dir\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534706    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-kubernetes\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55a8937d-e59e-4e64-9f56-4fe485e115f1-iptables-alerter-script\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534961    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-kubelet\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.534988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bkj\" (UniqueName: \"kubernetes.io/projected/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-kube-api-access-g9bkj\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-ovnkube-config\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-socket-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535006    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-cnibin\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba52014f-ffe4-4963-b240-5d8271d599a0-ovnkube-script-lib\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-log-socket\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-var-lib-kubelet\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-kubernetes\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e01dc35e-2818-4861-913c-f4df8df8f334-cnibin\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.535913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535130    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-cnibin\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-socket-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-netns\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-var-lib-kubelet\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-log-socket\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-ovn\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-cni-bin\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-ovn\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysctl-conf\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-cni-bin\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fr6\" (UniqueName: \"kubernetes.io/projected/e01dc35e-2818-4861-913c-f4df8df8f334-kube-api-access-69fr6\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-cni-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysctl-conf\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-kubelet\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-run-netns\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.536714 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-var-lib-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-run-netns\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-kubelet\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjrp\" (UniqueName: \"kubernetes.io/projected/ba52014f-ffe4-4963-b240-5d8271d599a0-kube-api-access-pwjrp\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-var-lib-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55a8937d-e59e-4e64-9f56-4fe485e115f1-iptables-alerter-script\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-system-cni-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/062d05c7-b02b-464f-abb6-3c6f113ee0ce-serviceca\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ftrt\" (UniqueName: \"kubernetes.io/projected/062d05c7-b02b-464f-abb6-3c6f113ee0ce-kube-api-access-2ftrt\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-systemd-units\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535734    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-systemd-units\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysconfig\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-host\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-registration-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8728adf3-07a6-4869-8405-6e7ba6a7ec6f-konnectivity-ca\") pod \"konnectivity-agent-c6vx2\" (UID: \"8728adf3-07a6-4869-8405-6e7ba6a7ec6f\") " pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13db78f9-f4eb-405b-b632-2a9eed835018-dbus\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.537617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-slash\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-etc-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-modprobe-d\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.535996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc89222e-825d-469d-96f7-9ef4ee6c15a6-registration-dir\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-systemd\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8728adf3-07a6-4869-8405-6e7ba6a7ec6f-agent-certs\") pod \"konnectivity-agent-c6vx2\" (UID: \"8728adf3-07a6-4869-8405-6e7ba6a7ec6f\") " pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-multus-certs\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-sysconfig\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-systemd\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/062d05c7-b02b-464f-abb6-3c6f113ee0ce-serviceca\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-cni-netd\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-host\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-modprobe-d\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-slash\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-cni-netd\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.538407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-run-systemd\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5efe24fd-9205-4070-805d-38ce688be1d8-etc-systemd\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba52014f-ffe4-4963-b240-5d8271d599a0-etc-openvswitch\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.536731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e01dc35e-2818-4861-913c-f4df8df8f334-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.537977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5efe24fd-9205-4070-805d-38ce688be1d8-etc-tuned\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.538586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba52014f-ffe4-4963-b240-5d8271d599a0-ovn-node-metrics-cert\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.539192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.538856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5efe24fd-9205-4070-805d-38ce688be1d8-tmp\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.542702 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.542498    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:26.542702 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.542526    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:26.542702 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.542540    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:26.542702 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.542625    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:27.042602588 +0000 UTC m=+2.951292944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:26.542702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.542624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4hw\" (UniqueName: \"kubernetes.io/projected/55a8937d-e59e-4e64-9f56-4fe485e115f1-kube-api-access-6c4hw\") pod \"iptables-alerter-cqgbv\" (UID: \"55a8937d-e59e-4e64-9f56-4fe485e115f1\") " pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.543618 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.543518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvtl\" (UniqueName: \"kubernetes.io/projected/5efe24fd-9205-4070-805d-38ce688be1d8-kube-api-access-7nvtl\") pod \"tuned-gcmzb\" (UID: \"5efe24fd-9205-4070-805d-38ce688be1d8\") " pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.543618 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.543561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqp26\" (UniqueName: \"kubernetes.io/projected/cc89222e-825d-469d-96f7-9ef4ee6c15a6-kube-api-access-bqp26\") pod \"aws-ebs-csi-driver-node-8rn5f\" (UID: \"cc89222e-825d-469d-96f7-9ef4ee6c15a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.544596 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.544556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjrp\" (UniqueName: \"kubernetes.io/projected/ba52014f-ffe4-4963-b240-5d8271d599a0-kube-api-access-pwjrp\") pod \"ovnkube-node-grsjn\" (UID: \"ba52014f-ffe4-4963-b240-5d8271d599a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.545850 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.545580    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fr6\" (UniqueName: \"kubernetes.io/projected/e01dc35e-2818-4861-913c-f4df8df8f334-kube-api-access-69fr6\") pod \"multus-additional-cni-plugins-bgxrn\" (UID: \"e01dc35e-2818-4861-913c-f4df8df8f334\") " pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.546163 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.546139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ftrt\" (UniqueName: \"kubernetes.io/projected/062d05c7-b02b-464f-abb6-3c6f113ee0ce-kube-api-access-2ftrt\") pod \"node-ca-hg8hl\" (UID: \"062d05c7-b02b-464f-abb6-3c6f113ee0ce\") " pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.571569 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.571516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal" event={"ID":"04c84e703d5815c2c5d803ec17e0971e","Type":"ContainerStarted","Data":"7fb2f202c2b0f9cc517f08bad27495fbfde3109c9d56e6f27b591637ee2d4ef6"}
Jul 29 22:46:26.572765 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.572739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal" event={"ID":"f65bcea48aa9eea90fe0167e5920afd3","Type":"ContainerStarted","Data":"22e36da667972a21566c3e497655cf2b5af1b316e7b821ecaef6bf8af8c24e81"}
Jul 29 22:46:26.637471 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-cnibin\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637471 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-netns\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-cni-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-netns\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-cnibin\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-system-cni-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.637624    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.637636    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8728adf3-07a6-4869-8405-6e7ba6a7ec6f-konnectivity-ca\") pod \"konnectivity-agent-c6vx2\" (UID: \"8728adf3-07a6-4869-8405-6e7ba6a7ec6f\") " pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-system-cni-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13db78f9-f4eb-405b-b632-2a9eed835018-dbus\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.637694 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.637688    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:27.137667785 +0000 UTC m=+3.046358149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13db78f9-f4eb-405b-b632-2a9eed835018-dbus\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-cni-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:26.637801    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:27.137763277 +0000 UTC m=+3.046453623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8728adf3-07a6-4869-8405-6e7ba6a7ec6f-agent-certs\") pod \"konnectivity-agent-c6vx2\" (UID: \"8728adf3-07a6-4869-8405-6e7ba6a7ec6f\") " pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-multus-certs\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twc5d\" (UniqueName: \"kubernetes.io/projected/af595ab0-8515-4923-af82-6cdcd20b5c57-kube-api-access-twc5d\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbbk\" (UniqueName: \"kubernetes.io/projected/df0d9576-7662-40f9-b242-50eb3acada25-kube-api-access-mdbbk\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-socket-dir-parent\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-multus-certs\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-cni-multus\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.637994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-conf-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-os-release\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-cni-binary-copy\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-daemon-config\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0d9576-7662-40f9-b242-50eb3acada25-tmp-dir\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-hostroot\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.638297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df0d9576-7662-40f9-b242-50eb3acada25-hosts-file\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638161    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-etc-kubernetes\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-k8s-cni-cncf-io\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-cni-bin\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13db78f9-f4eb-405b-b632-2a9eed835018-kubelet-config\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-kubelet\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bkj\" (UniqueName: \"kubernetes.io/projected/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-kube-api-access-g9bkj\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-cni-multus\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8728adf3-07a6-4869-8405-6e7ba6a7ec6f-konnectivity-ca\") pod \"konnectivity-agent-c6vx2\" (UID: \"8728adf3-07a6-4869-8405-6e7ba6a7ec6f\") " pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df0d9576-7662-40f9-b242-50eb3acada25-hosts-file\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-socket-dir-parent\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-etc-kubernetes\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-cni-bin\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638450    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13db78f9-f4eb-405b-b632-2a9eed835018-kubelet-config\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-os-release\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638666    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-hostroot\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-run-k8s-cni-cncf-io\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-cni-binary-copy\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.639253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-host-var-lib-kubelet\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.640121 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638773    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-conf-dir\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.640121 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df0d9576-7662-40f9-b242-50eb3acada25-tmp-dir\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.640121 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.638898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-multus-daemon-config\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.640642 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.640626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8728adf3-07a6-4869-8405-6e7ba6a7ec6f-agent-certs\") pod \"konnectivity-agent-c6vx2\" (UID: \"8728adf3-07a6-4869-8405-6e7ba6a7ec6f\") " pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.647173 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.647149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twc5d\" (UniqueName: \"kubernetes.io/projected/af595ab0-8515-4923-af82-6cdcd20b5c57-kube-api-access-twc5d\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:26.647296 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.647196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bkj\" (UniqueName: \"kubernetes.io/projected/95ffbc47-f716-43e4-96f4-7f4f97ee47ed-kube-api-access-g9bkj\") pod \"multus-p94q8\" (UID: \"95ffbc47-f716-43e4-96f4-7f4f97ee47ed\") " pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.647296 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.647217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbbk\" (UniqueName: \"kubernetes.io/projected/df0d9576-7662-40f9-b242-50eb3acada25-kube-api-access-mdbbk\") pod \"node-resolver-ts59j\" (UID: \"df0d9576-7662-40f9-b242-50eb3acada25\") " pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:26.726127 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.726042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hg8hl"
Jul 29 22:46:26.734894 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.734867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:26.745886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.745862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cqgbv"
Jul 29 22:46:26.752649 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.752625    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gcmzb"
Jul 29 22:46:26.759380 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.759357    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f"
Jul 29 22:46:26.762617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.762600    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 22:46:26.769092 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.769074    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bgxrn"
Jul 29 22:46:26.774713 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.774696    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:26.782309 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.782294    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p94q8"
Jul 29 22:46:26.787678 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:26.787654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ts59j"
Jul 29 22:46:27.141248 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.141215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:27.141248 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.141253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:27.141467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.141273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:27.141467 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141400    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:27.141467 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141422    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:27.141467 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141441    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:27.141467 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141450    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:27.141467 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141467    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:28.141442888 +0000 UTC m=+4.050133237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:27.141681 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141399    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:27.141681 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141493    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:28.141479216 +0000 UTC m=+4.050169571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:27.141681 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.141555    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:28.141533128 +0000 UTC m=+4.050223467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:27.461439 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.461348    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 22:41:26 +0000 UTC" deadline="2028-01-06 19:16:14.788151479 +0000 UTC"
Jul 29 22:46:27.461439 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.461387    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12620h29m47.326769763s"
Jul 29 22:46:27.519176 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.519145    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ffbc47_f716_43e4_96f4_7f4f97ee47ed.slice/crio-8011cfbe22637f2a6e6cfa37a4e647c70afe0411b7c88683f9f49d93a3d407e3 WatchSource:0}: Error finding container 8011cfbe22637f2a6e6cfa37a4e647c70afe0411b7c88683f9f49d93a3d407e3: Status 404 returned error can't find the container with id 8011cfbe22637f2a6e6cfa37a4e647c70afe0411b7c88683f9f49d93a3d407e3
Jul 29 22:46:27.522555 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.522526    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01dc35e_2818_4861_913c_f4df8df8f334.slice/crio-15eb67f921f8c4dc0ac5081bd2c62c0e79654f2ee00f545bebde6b27adf5a0f7 WatchSource:0}: Error finding container 15eb67f921f8c4dc0ac5081bd2c62c0e79654f2ee00f545bebde6b27adf5a0f7: Status 404 returned error can't find the container with id 15eb67f921f8c4dc0ac5081bd2c62c0e79654f2ee00f545bebde6b27adf5a0f7
Jul 29 22:46:27.524660 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.524625    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062d05c7_b02b_464f_abb6_3c6f113ee0ce.slice/crio-59e66867e6e6261b0e3b69d977b3393a3c881cc6a46b7bf35eec16c0445e3605 WatchSource:0}: Error finding container 59e66867e6e6261b0e3b69d977b3393a3c881cc6a46b7bf35eec16c0445e3605: Status 404 returned error can't find the container with id 59e66867e6e6261b0e3b69d977b3393a3c881cc6a46b7bf35eec16c0445e3605
Jul 29 22:46:27.525777 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.525752    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efe24fd_9205_4070_805d_38ce688be1d8.slice/crio-bc53119a369828b28fdc3094ec62560e8c8147a6126225cc79312689db2259d6 WatchSource:0}: Error finding container bc53119a369828b28fdc3094ec62560e8c8147a6126225cc79312689db2259d6: Status 404 returned error can't find the container with id bc53119a369828b28fdc3094ec62560e8c8147a6126225cc79312689db2259d6
Jul 29 22:46:27.527038 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.527015    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a8937d_e59e_4e64_9f56_4fe485e115f1.slice/crio-197eb10cc1cc77b79f5a571e2f22acf9e498223d5b14336ed69a90bd7b08f455 WatchSource:0}: Error finding container 197eb10cc1cc77b79f5a571e2f22acf9e498223d5b14336ed69a90bd7b08f455: Status 404 returned error can't find the container with id 197eb10cc1cc77b79f5a571e2f22acf9e498223d5b14336ed69a90bd7b08f455
Jul 29 22:46:27.527848 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.527828    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc89222e_825d_469d_96f7_9ef4ee6c15a6.slice/crio-23ef928df13fe789765f60abd4ffd188f6f4651921382802881fa0c5b6dedde4 WatchSource:0}: Error finding container 23ef928df13fe789765f60abd4ffd188f6f4651921382802881fa0c5b6dedde4: Status 404 returned error can't find the container with id 23ef928df13fe789765f60abd4ffd188f6f4651921382802881fa0c5b6dedde4
Jul 29 22:46:27.529121 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.529024    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8728adf3_07a6_4869_8405_6e7ba6a7ec6f.slice/crio-2aab6d0e9508eaf81cb417d048362b7c4c718467f0419983ffc733e7440843d0 WatchSource:0}: Error finding container 2aab6d0e9508eaf81cb417d048362b7c4c718467f0419983ffc733e7440843d0: Status 404 returned error can't find the container with id 2aab6d0e9508eaf81cb417d048362b7c4c718467f0419983ffc733e7440843d0
Jul 29 22:46:27.529812 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.529768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0d9576_7662_40f9_b242_50eb3acada25.slice/crio-c94b0db4c1d8453ccf6c89151149526daed93870a7a39b7a5cd64f59d3b501a5 WatchSource:0}: Error finding container c94b0db4c1d8453ccf6c89151149526daed93870a7a39b7a5cd64f59d3b501a5: Status 404 returned error can't find the container with id c94b0db4c1d8453ccf6c89151149526daed93870a7a39b7a5cd64f59d3b501a5
Jul 29 22:46:27.531603 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:27.531582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba52014f_ffe4_4963_b240_5d8271d599a0.slice/crio-6979a9eaca3cd25a46ed060213177f00ef31e9fc3b5d4782b58e57bd888070e7 WatchSource:0}: Error finding container 6979a9eaca3cd25a46ed060213177f00ef31e9fc3b5d4782b58e57bd888070e7: Status 404 returned error can't find the container with id 6979a9eaca3cd25a46ed060213177f00ef31e9fc3b5d4782b58e57bd888070e7
Jul 29 22:46:27.563577 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.563558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:27.563664 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.563581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:27.563664 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.563654    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:27.563741 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:27.563717    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:27.575697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.575669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hg8hl" event={"ID":"062d05c7-b02b-464f-abb6-3c6f113ee0ce","Type":"ContainerStarted","Data":"59e66867e6e6261b0e3b69d977b3393a3c881cc6a46b7bf35eec16c0445e3605"}
Jul 29 22:46:27.576839 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.576815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c6vx2" event={"ID":"8728adf3-07a6-4869-8405-6e7ba6a7ec6f","Type":"ContainerStarted","Data":"2aab6d0e9508eaf81cb417d048362b7c4c718467f0419983ffc733e7440843d0"}
Jul 29 22:46:27.577871 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.577847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f" event={"ID":"cc89222e-825d-469d-96f7-9ef4ee6c15a6","Type":"ContainerStarted","Data":"23ef928df13fe789765f60abd4ffd188f6f4651921382802881fa0c5b6dedde4"}
Jul 29 22:46:27.578803 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.578766    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cqgbv" event={"ID":"55a8937d-e59e-4e64-9f56-4fe485e115f1","Type":"ContainerStarted","Data":"197eb10cc1cc77b79f5a571e2f22acf9e498223d5b14336ed69a90bd7b08f455"}
Jul 29 22:46:27.579835 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.579813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gcmzb" event={"ID":"5efe24fd-9205-4070-805d-38ce688be1d8","Type":"ContainerStarted","Data":"bc53119a369828b28fdc3094ec62560e8c8147a6126225cc79312689db2259d6"}
Jul 29 22:46:27.580994 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.580858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerStarted","Data":"15eb67f921f8c4dc0ac5081bd2c62c0e79654f2ee00f545bebde6b27adf5a0f7"}
Jul 29 22:46:27.582107 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.582090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p94q8" event={"ID":"95ffbc47-f716-43e4-96f4-7f4f97ee47ed","Type":"ContainerStarted","Data":"8011cfbe22637f2a6e6cfa37a4e647c70afe0411b7c88683f9f49d93a3d407e3"}
Jul 29 22:46:27.583222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.583199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"6979a9eaca3cd25a46ed060213177f00ef31e9fc3b5d4782b58e57bd888070e7"}
Jul 29 22:46:27.586357 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:27.586333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ts59j" event={"ID":"df0d9576-7662-40f9-b242-50eb3acada25","Type":"ContainerStarted","Data":"c94b0db4c1d8453ccf6c89151149526daed93870a7a39b7a5cd64f59d3b501a5"}
Jul 29 22:46:28.150527 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.150489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:28.150713 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.150546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:28.150713 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.150574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:28.150858 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.150764    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:28.150858 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.150799    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:28.150858 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.150812    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:28.150990 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.150874    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:30.150854309 +0000 UTC m=+6.059544665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:28.151532 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.151347    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:28.151532 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.151407    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:30.151387811 +0000 UTC m=+6.060078150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:28.151532 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.151466    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:28.151532 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.151496    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:30.151485756 +0000 UTC m=+6.060176097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:28.462269 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.462123    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 22:41:26 +0000 UTC" deadline="2028-05-07 02:11:55.79844831 +0000 UTC"
Jul 29 22:46:28.462269 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.462189    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15531h25m27.336263971s"
Jul 29 22:46:28.566862 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.566377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:28.566862 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:28.566534    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:28.597156 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.597111    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal" event={"ID":"04c84e703d5815c2c5d803ec17e0971e","Type":"ContainerStarted","Data":"5702426d988d18a37c528959ae3044450a20da5c3342f804de3b544fccb53814"}
Jul 29 22:46:28.605349 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.605259    2580 generic.go:356] "Generic (PLEG): container finished" podID="f65bcea48aa9eea90fe0167e5920afd3" containerID="6ddfa196ae82a0262f69657e33519010f27e4851d6e871d2fd584795140ebc8f" exitCode=0
Jul 29 22:46:28.605349 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.605305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal" event={"ID":"f65bcea48aa9eea90fe0167e5920afd3","Type":"ContainerDied","Data":"6ddfa196ae82a0262f69657e33519010f27e4851d6e871d2fd584795140ebc8f"}
Jul 29 22:46:28.628529 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:28.628487    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-193.ec2.internal" podStartSLOduration=3.628471437 podStartE2EDuration="3.628471437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:28.612307636 +0000 UTC m=+4.520997994" watchObservedRunningTime="2026-07-29 22:46:28.628471437 +0000 UTC m=+4.537161796"
Jul 29 22:46:29.564079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:29.564044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:29.564562 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:29.564181    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:29.564629 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:29.564618    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:29.564740 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:29.564715    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:29.631858 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:29.631340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal" event={"ID":"f65bcea48aa9eea90fe0167e5920afd3","Type":"ContainerStarted","Data":"53dd3f0248985e008e5cb3b3b621f2b1c976ebe4722d01464ca45ed469151262"}
Jul 29 22:46:30.174274 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:30.174236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:30.174477 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:30.174300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:30.174477 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:30.174329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:30.174591 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.174491    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:30.174591 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.174512    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:30.174591 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.174525    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:30.174591 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.174582    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:34.17456433 +0000 UTC m=+10.083254682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:30.174992 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.174974    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:30.175076 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.175029    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:34.175012753 +0000 UTC m=+10.083703096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:30.175142 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.175085    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:30.175142 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.175119    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:34.17510801 +0000 UTC m=+10.083798350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:30.565989 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:30.565480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:30.565989 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:30.565615    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:31.564213 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:31.564181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:31.564213 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:31.564206    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:31.564452 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:31.564327    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:31.564834 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:31.564742    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:32.563550 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:32.563513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:32.564007 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:32.563687    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:33.564580 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:33.564157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:33.564580 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:33.564273    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:33.564580 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:33.564366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:33.564580 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:33.564548    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:34.208063 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:34.208025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:34.208253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:34.208073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:34.208253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:34.208103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:34.208253 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208199    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:34.208403 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:42.208244572 +0000 UTC m=+18.116934916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:34.208403 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208279    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:34.208403 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208297    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:34.208403 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208310    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:34.208403 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208376    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:34.208403 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208393    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:42.208376999 +0000 UTC m=+18.117067349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:34.208674 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.208418    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:42.208408219 +0000 UTC m=+18.117098559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:34.564949 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:34.564573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:34.564949 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:34.564708    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:35.563406 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:35.563361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:35.563575 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:35.563508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:35.563983 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:35.563375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:35.563983 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:35.563883    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:36.564315 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:36.564276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:36.564755 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:36.564435    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:37.563884 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:37.563859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:37.564079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:37.563859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:37.564079 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:37.563971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:37.564079 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:37.564036    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:38.564311 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:38.564271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:38.564762 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:38.564432    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:39.564112 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:39.564076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:39.564279 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:39.564076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:39.564279 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:39.564196    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:39.564618 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:39.564282    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:40.564297 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:40.564254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:40.564481 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:40.564386    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:41.564135 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:41.564099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:41.564338 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:41.564099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:41.564338 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:41.564209    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:41.564338 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:41.564302    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:42.268779 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:42.268740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:42.268805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:42.268836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.268907    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.268956    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.268972    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.268981    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.268907    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.268981    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:58.268967049 +0000 UTC m=+34.177657405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.269061    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:58.269044156 +0000 UTC m=+34.177734501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:42.269336 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.269078    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:58.269068478 +0000 UTC m=+34.177758820 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:42.563567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:42.563532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:42.563758 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:42.563659    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:43.563621 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:43.563590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:43.564036 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:43.563696    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:43.564036 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:43.563776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:43.564036 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:43.563889    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:44.565120 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:44.565078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:44.565563 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:44.565213    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:45.564104 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:45.564063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:45.564267 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:45.564063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:45.564267 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:45.564179    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:45.564267 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:45.564236    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:46.564724 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.564300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:46.565479 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:46.564874    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:46.668507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.668415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c6vx2" event={"ID":"8728adf3-07a6-4869-8405-6e7ba6a7ec6f","Type":"ContainerStarted","Data":"3a9026f7dc93906a7d2d6f17300b744cbb2b345b1a62ddb27726f1e8caf0edee"}
Jul 29 22:46:46.670262 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.670229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f" event={"ID":"cc89222e-825d-469d-96f7-9ef4ee6c15a6","Type":"ContainerStarted","Data":"8a621715e1e48ca048c52587c7db9dd9c46138bf50672cca3a082f91a84f9d1e"}
Jul 29 22:46:46.671910 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.671884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gcmzb" event={"ID":"5efe24fd-9205-4070-805d-38ce688be1d8","Type":"ContainerStarted","Data":"792a5cec29dfd802c13ababe736c189c517f35ee196631f02931fedf904f91db"}
Jul 29 22:46:46.673552 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.673520    2580 generic.go:356] "Generic (PLEG): container finished" podID="e01dc35e-2818-4861-913c-f4df8df8f334" containerID="721fadcb9faafbffd70d546b66426a6279973abf746db04c2a74daebbf1cebc1" exitCode=0
Jul 29 22:46:46.673664 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.673592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerDied","Data":"721fadcb9faafbffd70d546b66426a6279973abf746db04c2a74daebbf1cebc1"}
Jul 29 22:46:46.675569 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.675171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p94q8" event={"ID":"95ffbc47-f716-43e4-96f4-7f4f97ee47ed","Type":"ContainerStarted","Data":"5a71fa93bdea91a39110d65817770af6af3318767e1f167334ad3cb857099d68"}
Jul 29 22:46:46.678462 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.678438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"f4eb7f827b710c6c3e8cc96a0ad203b49bae9567b52ad9139c85075b59dc858d"}
Jul 29 22:46:46.678597 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.678569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"bc1dca790a2a8a4d0b51337f388e4ebf7ebabaed9096a75305ba953d6f2539e9"}
Jul 29 22:46:46.678694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.678635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"f8a753c5dcd332a5ab6bca475c6db0759ea1c08db65b6041e84f96fa6446b3c1"}
Jul 29 22:46:46.678694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.678685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"6ab1ad62d6cb55cdf7aa55d91229efc355ee93eb153e27189046c11d3e759180"}
Jul 29 22:46:46.678823 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.678733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"ba3c0c8d7ff1d287b3d936890aa2d02a69c05fe2d61e5058975de806a4be5ac1"}
Jul 29 22:46:46.678823 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.678781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"14866d3ee91a11cd5fd23d98c66120652d14ce6a3c2eb695609f2688940c63b1"}
Jul 29 22:46:46.680216 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.680183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ts59j" event={"ID":"df0d9576-7662-40f9-b242-50eb3acada25","Type":"ContainerStarted","Data":"a9a6108c46fdf96f98fc7ad146674d117dce6ce2478cd66ea6aeb0ae53638d86"}
Jul 29 22:46:46.681660 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.681618    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hg8hl" event={"ID":"062d05c7-b02b-464f-abb6-3c6f113ee0ce","Type":"ContainerStarted","Data":"e9c90c0543ae9391c669fc37b3317285796ef1c7ad5445df3873c59d079dca51"}
Jul 29 22:46:46.682863 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.682769    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-193.ec2.internal" podStartSLOduration=21.682757644 podStartE2EDuration="21.682757644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:29.647543859 +0000 UTC m=+5.556234222" watchObservedRunningTime="2026-07-29 22:46:46.682757644 +0000 UTC m=+22.591448003"
Jul 29 22:46:46.683269 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.683241    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c6vx2" podStartSLOduration=8.677829282 podStartE2EDuration="22.683232038s" totalImagesPullingTime="14.005402756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:46.681905552 +0000 UTC m=+22.590595909" watchObservedRunningTime="2026-07-29 22:46:46.683232038 +0000 UTC m=+22.591922395"
Jul 29 22:46:46.711456 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.711414    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hg8hl" podStartSLOduration=4.393902045 podStartE2EDuration="22.711402091s" totalImagesPullingTime="18.317500046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:46.694030995 +0000 UTC m=+22.602721354" watchObservedRunningTime="2026-07-29 22:46:46.711402091 +0000 UTC m=+22.620092448"
Jul 29 22:46:46.726360 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.726324    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p94q8" podStartSLOduration=4.340348905 podStartE2EDuration="22.7263123s" totalImagesPullingTime="18.385963395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:46.725690599 +0000 UTC m=+22.634380956" watchObservedRunningTime="2026-07-29 22:46:46.7263123 +0000 UTC m=+22.635002657"
Jul 29 22:46:46.739663 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.739624    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gcmzb" podStartSLOduration=4.397577067 podStartE2EDuration="22.739611275s" totalImagesPullingTime="18.342034208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:46.739558234 +0000 UTC m=+22.648248592" watchObservedRunningTime="2026-07-29 22:46:46.739611275 +0000 UTC m=+22.648301632"
Jul 29 22:46:46.753467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:46.753419    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ts59j" podStartSLOduration=4.436536443 podStartE2EDuration="22.753407463s" totalImagesPullingTime="18.31687102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:46.752878324 +0000 UTC m=+22.661568684" watchObservedRunningTime="2026-07-29 22:46:46.753407463 +0000 UTC m=+22.662097820"
Jul 29 22:46:47.303091 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.303067    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 22:46:47.495952 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.495801    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T22:46:47.303085373Z","UUID":"2fb5990a-bea8-4caa-8ef3-05d47b7f3d4f","Handler":null,"Name":"","Endpoint":""}
Jul 29 22:46:47.497203 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.497187    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 22:46:47.497303 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.497221    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 22:46:47.563744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.563719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:47.563929 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.563725    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:47.563929 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:47.563839    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:47.564003 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:47.563926    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:47.675806 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.675763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:47.676423 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.676408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:47.686211 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.686171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f" event={"ID":"cc89222e-825d-469d-96f7-9ef4ee6c15a6","Type":"ContainerStarted","Data":"56bc2043e934b92c952491a078a450091e505cac2d26948cf624531a2d1506e2"}
Jul 29 22:46:47.687819 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.687744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cqgbv" event={"ID":"55a8937d-e59e-4e64-9f56-4fe485e115f1","Type":"ContainerStarted","Data":"63c4cc9715efd4cc7731a040585cbab222b41b808b95aca756269bfa45fc50d2"}
Jul 29 22:46:47.688550 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.688522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:47.688671 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.688594    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 22:46:47.720133 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:47.720079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cqgbv" podStartSLOduration=5.405288308 podStartE2EDuration="23.72005865s" totalImagesPullingTime="18.314770342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:47.71933945 +0000 UTC m=+23.628029808" watchObservedRunningTime="2026-07-29 22:46:47.72005865 +0000 UTC m=+23.628749010"
Jul 29 22:46:48.563873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:48.563844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:48.564054 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:48.563978    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:48.692355 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:48.692319    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f" event={"ID":"cc89222e-825d-469d-96f7-9ef4ee6c15a6","Type":"ContainerStarted","Data":"ca48453416b044bf34378534118bd4f125865806475e263682270ac6eb049ef9"}
Jul 29 22:46:48.711126 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:48.711078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8rn5f" podStartSLOduration=4.118810543 podStartE2EDuration="24.711059899s" totalImagesPullingTime="20.592249356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:48.710776138 +0000 UTC m=+24.619466495" watchObservedRunningTime="2026-07-29 22:46:48.711059899 +0000 UTC m=+24.619750257"
Jul 29 22:46:49.564225 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:49.564197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:49.564225 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:49.564215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:49.564484 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:49.564305    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:49.564484 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:49.564469    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:49.697954 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:49.697917    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"2cda37309b52cc1cf5a31d05777e694c96dbf84c636ec4cb3b86d903be2d2a00"}
Jul 29 22:46:50.564550 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:50.564346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:50.564801 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:50.564653    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:51.564121 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.564085    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:51.564121 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.564116    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:51.564811 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:51.564193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:51.564811 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:51.564313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:51.704497 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.704424    2580 generic.go:356] "Generic (PLEG): container finished" podID="e01dc35e-2818-4861-913c-f4df8df8f334" containerID="948c41f350c3a553b992fa82fba08efa4f27a6655790267d4a1b778d76fc6912" exitCode=0
Jul 29 22:46:51.704627 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.704504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerDied","Data":"948c41f350c3a553b992fa82fba08efa4f27a6655790267d4a1b778d76fc6912"}
Jul 29 22:46:51.708108 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.708080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" event={"ID":"ba52014f-ffe4-4963-b240-5d8271d599a0","Type":"ContainerStarted","Data":"8c18281ba80c097ba502dd00cec0d006e53a7378311f93d2e281773162bf12c0"}
Jul 29 22:46:51.708347 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.708314    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:51.708441 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.708392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:51.708508 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.708448    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:51.723316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.723157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:51.723456 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.723436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:46:51.746525 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:51.746478    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn" podStartSLOduration=9.364035791 podStartE2EDuration="27.746462459s" totalImagesPullingTime="18.382426668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:46:51.746438552 +0000 UTC m=+27.655128910" watchObservedRunningTime="2026-07-29 22:46:51.746462459 +0000 UTC m=+27.655152807"
Jul 29 22:46:52.564326 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:52.564296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:52.564865 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:52.564441    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:53.326957 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.326918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8j5c"]
Jul 29 22:46:53.327128 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.327051    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:53.327195 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:53.327149    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:53.329946 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.329921    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2psct"]
Jul 29 22:46:53.330088 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.330024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:53.330154 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:53.330119    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:53.330531 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.330507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vct9g"]
Jul 29 22:46:53.330628 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.330604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:53.330778 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:53.330717    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:53.715220 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.715189    2580 generic.go:356] "Generic (PLEG): container finished" podID="e01dc35e-2818-4861-913c-f4df8df8f334" containerID="4d766baf4d3983fe9a85fe722f265c309b0345d520764daa00a52d56b6895128" exitCode=0
Jul 29 22:46:53.715682 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:53.715262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerDied","Data":"4d766baf4d3983fe9a85fe722f265c309b0345d520764daa00a52d56b6895128"}
Jul 29 22:46:55.564402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:55.564231    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:55.564859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:55.564272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:55.564859 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:55.564490    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:55.564859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:55.564272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:55.564859 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:55.564564    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:55.564859 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:55.564659    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:55.721876 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:55.721844    2580 generic.go:356] "Generic (PLEG): container finished" podID="e01dc35e-2818-4861-913c-f4df8df8f334" containerID="47dd0ef205fdc6fca80f66debcbd08834b841e0c9ab8f1a8f90b46df4aef8912" exitCode=0
Jul 29 22:46:55.722065 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:55.721904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerDied","Data":"47dd0ef205fdc6fca80f66debcbd08834b841e0c9ab8f1a8f90b46df4aef8912"}
Jul 29 22:46:57.564062 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:57.564028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:57.564549 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:57.564156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:57.564549 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:57.564164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2psct" podUID="8a089a4c-8e5a-4878-849a-bbf19f441983"
Jul 29 22:46:57.564549 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:57.564190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:57.564549 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:57.564294    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8j5c" podUID="13db78f9-f4eb-405b-b632-2a9eed835018"
Jul 29 22:46:57.564549 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:57.564372    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:46:58.288555 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.288515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:58.288737 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.288565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:58.288737 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.288594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:58.288737 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288676    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:58.288737 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288708    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:58.288737 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288717    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 22:46:58.288737 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288738    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 22:46:58.289104 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288751    2580 projected.go:196] Error preparing data for projected volume kube-api-access-nwtk6 for pod openshift-network-diagnostics/network-check-target-2psct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:58.289104 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:30.288730667 +0000 UTC m=+66.197421009 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 22:46:58.289104 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288827    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret podName:13db78f9-f4eb-405b-b632-2a9eed835018 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:30.288811283 +0000 UTC m=+66.197501624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret") pod "global-pull-secret-syncer-x8j5c" (UID: "13db78f9-f4eb-405b-b632-2a9eed835018") : object "kube-system"/"original-pull-secret" not registered
Jul 29 22:46:58.289104 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.288842    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6 podName:8a089a4c-8e5a-4878-849a-bbf19f441983 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:30.288832426 +0000 UTC m=+66.197522769 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwtk6" (UniqueName: "kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6") pod "network-check-target-2psct" (UID: "8a089a4c-8e5a-4878-849a-bbf19f441983") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 22:46:58.384312 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.384288    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-193.ec2.internal" event="NodeReady"
Jul 29 22:46:58.384470 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.384454    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 22:46:58.429259 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.429229    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wt4mt"]
Jul 29 22:46:58.431988 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.431924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.432129 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.432038    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4hpqx"]
Jul 29 22:46:58.433941 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.433919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.434211 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.434185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtttq\""
Jul 29 22:46:58.434440 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.434419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 22:46:58.434440 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.434429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 22:46:58.435939 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.435810    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 22:46:58.436041 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.436007    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 22:46:58.436127 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.436109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 22:46:58.436575 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.436524    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d8h7n\""
Jul 29 22:46:58.442027 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.441779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wt4mt"]
Jul 29 22:46:58.444733 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.444714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4hpqx"]
Jul 29 22:46:58.590800 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.590755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.591337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.590812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00a5e172-5fa6-4f97-b08b-33d512358027-cert\") pod \"ingress-canary-4hpqx\" (UID: \"00a5e172-5fa6-4f97-b08b-33d512358027\") " pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.591337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.590839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c721bef-22cf-40a8-878c-f071e3a1f012-tmp-dir\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.591337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.590925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4l5\" (UniqueName: \"kubernetes.io/projected/7c721bef-22cf-40a8-878c-f071e3a1f012-kube-api-access-gz4l5\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.591337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.591018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c721bef-22cf-40a8-878c-f071e3a1f012-config-volume\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.591337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.591062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48hp\" (UniqueName: \"kubernetes.io/projected/00a5e172-5fa6-4f97-b08b-33d512358027-kube-api-access-q48hp\") pod \"ingress-canary-4hpqx\" (UID: \"00a5e172-5fa6-4f97-b08b-33d512358027\") " pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.691674 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.691640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48hp\" (UniqueName: \"kubernetes.io/projected/00a5e172-5fa6-4f97-b08b-33d512358027-kube-api-access-q48hp\") pod \"ingress-canary-4hpqx\" (UID: \"00a5e172-5fa6-4f97-b08b-33d512358027\") " pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.691674 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.691678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.691922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.691704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00a5e172-5fa6-4f97-b08b-33d512358027-cert\") pod \"ingress-canary-4hpqx\" (UID: \"00a5e172-5fa6-4f97-b08b-33d512358027\") " pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.691922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.691723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c721bef-22cf-40a8-878c-f071e3a1f012-tmp-dir\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.691922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.691759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4l5\" (UniqueName: \"kubernetes.io/projected/7c721bef-22cf-40a8-878c-f071e3a1f012-kube-api-access-gz4l5\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.691922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.691843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c721bef-22cf-40a8-878c-f071e3a1f012-config-volume\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.691922 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.691901    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:58.692160 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:58.691973    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:46:59.191957279 +0000 UTC m=+35.100647615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:46:58.692160 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.692102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c721bef-22cf-40a8-878c-f071e3a1f012-tmp-dir\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.692893 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.692871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c721bef-22cf-40a8-878c-f071e3a1f012-config-volume\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.696536 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.696515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00a5e172-5fa6-4f97-b08b-33d512358027-cert\") pod \"ingress-canary-4hpqx\" (UID: \"00a5e172-5fa6-4f97-b08b-33d512358027\") " pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.699896 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.699872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4l5\" (UniqueName: \"kubernetes.io/projected/7c721bef-22cf-40a8-878c-f071e3a1f012-kube-api-access-gz4l5\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:58.700027 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.699996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48hp\" (UniqueName: \"kubernetes.io/projected/00a5e172-5fa6-4f97-b08b-33d512358027-kube-api-access-q48hp\") pod \"ingress-canary-4hpqx\" (UID: \"00a5e172-5fa6-4f97-b08b-33d512358027\") " pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:58.751598 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:58.751568    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4hpqx"
Jul 29 22:46:59.195719 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.195687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:46:59.195928 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:59.195831    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:46:59.195928 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:46:59.195902    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:00.195882797 +0000 UTC m=+36.104573136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:46:59.563818 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.563778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:46:59.563973 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.563778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:46:59.563973 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.563778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:46:59.566316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.566299    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 22:46:59.566429 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.566298    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 22:46:59.566916 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.566895    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 22:46:59.566916 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.566906    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 22:46:59.567079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.566952    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csbtv\""
Jul 29 22:46:59.567079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.567006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkbjs\""
Jul 29 22:46:59.702270 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.702089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4hpqx"]
Jul 29 22:46:59.705241 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:46:59.705215    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a5e172_5fa6_4f97_b08b_33d512358027.slice/crio-f02e1a83379ca188a6ea5cdbd5366053b2b5ad9eff5b9b70ce1441dbaa372a6e WatchSource:0}: Error finding container f02e1a83379ca188a6ea5cdbd5366053b2b5ad9eff5b9b70ce1441dbaa372a6e: Status 404 returned error can't find the container with id f02e1a83379ca188a6ea5cdbd5366053b2b5ad9eff5b9b70ce1441dbaa372a6e
Jul 29 22:46:59.737092 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.737062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerStarted","Data":"d0905d94433a107f6c6fe7debc5f83eb37c80430670ab2bdc09b5b9ade848b9e"}
Jul 29 22:46:59.738169 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:46:59.738139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4hpqx" event={"ID":"00a5e172-5fa6-4f97-b08b-33d512358027","Type":"ContainerStarted","Data":"f02e1a83379ca188a6ea5cdbd5366053b2b5ad9eff5b9b70ce1441dbaa372a6e"}
Jul 29 22:47:00.202128 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:00.202092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:47:00.202294 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:00.202266    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:00.202343 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:00.202337    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:02.202314819 +0000 UTC m=+38.111005164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:00.744587 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:00.744548    2580 generic.go:356] "Generic (PLEG): container finished" podID="e01dc35e-2818-4861-913c-f4df8df8f334" containerID="d0905d94433a107f6c6fe7debc5f83eb37c80430670ab2bdc09b5b9ade848b9e" exitCode=0
Jul 29 22:47:00.745075 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:00.744636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerDied","Data":"d0905d94433a107f6c6fe7debc5f83eb37c80430670ab2bdc09b5b9ade848b9e"}
Jul 29 22:47:01.749573 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:01.749545    2580 generic.go:356] "Generic (PLEG): container finished" podID="e01dc35e-2818-4861-913c-f4df8df8f334" containerID="d2433650c51211e9850b5eced33cb5040e4d71b4efee62fd1d1daa455c827c37" exitCode=0
Jul 29 22:47:01.750036 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:01.749596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerDied","Data":"d2433650c51211e9850b5eced33cb5040e4d71b4efee62fd1d1daa455c827c37"}
Jul 29 22:47:02.218327 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:02.218137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:47:02.218327 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:02.218312    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:02.218567 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:02.218393    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:06.218371682 +0000 UTC m=+42.127062019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:02.753571 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:02.753535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4hpqx" event={"ID":"00a5e172-5fa6-4f97-b08b-33d512358027","Type":"ContainerStarted","Data":"1430da261e00670f0eaf8cbb53efb6b86e2a36e55ec36a5d79e56eec2174c426"}
Jul 29 22:47:02.756809 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:02.756769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" event={"ID":"e01dc35e-2818-4861-913c-f4df8df8f334","Type":"ContainerStarted","Data":"b52a5cec72285243cfdd65663d1bf2264f5618ec0e3f66f008c173fa5de2eaeb"}
Jul 29 22:47:02.768393 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:02.768339    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4hpqx" podStartSLOduration=2.571477423 podStartE2EDuration="4.768320607s" totalImagesPullingTime="2.196843184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:02.76811979 +0000 UTC m=+38.676810149" watchObservedRunningTime="2026-07-29 22:47:02.768320607 +0000 UTC m=+38.677010966"
Jul 29 22:47:02.789001 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:02.788952    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bgxrn" podStartSLOduration=9.858058614 podStartE2EDuration="38.788935173s" totalImagesPullingTime="28.930876559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:02.788328938 +0000 UTC m=+38.697019308" watchObservedRunningTime="2026-07-29 22:47:02.788935173 +0000 UTC m=+38.697625530"
Jul 29 22:47:06.247917 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:06.247876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:47:06.248350 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:06.248014    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:06.248350 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:06.248070    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:14.248053276 +0000 UTC m=+50.156743617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:14.301184 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:14.301137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:47:14.301777 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:14.301271    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:14.301777 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:14.301345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:47:30.301323138 +0000 UTC m=+66.210013478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:23.733311 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:23.733281    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grsjn"
Jul 29 22:47:30.308576 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.308540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:47:30.308576 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.308581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:47:30.309079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.308600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:47:30.309079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.308632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:47:30.309079 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:30.308714    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:47:30.309079 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:30.308842    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:48:02.308779695 +0000 UTC m=+98.217470045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:47:30.310961 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.310942    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 22:47:30.311202 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.311188    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 22:47:30.311252 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.311203    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 22:47:30.319557 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:30.319537    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:47:30.319603 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:47:30.319589    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:48:34.319574775 +0000 UTC m=+130.228265111 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : secret "metrics-daemon-secret" not found
Jul 29 22:47:30.321374 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.321353    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 22:47:30.323383 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.323363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13db78f9-f4eb-405b-b632-2a9eed835018-original-pull-secret\") pod \"global-pull-secret-syncer-x8j5c\" (UID: \"13db78f9-f4eb-405b-b632-2a9eed835018\") " pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:47:30.332663 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.332640    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtk6\" (UniqueName: \"kubernetes.io/projected/8a089a4c-8e5a-4878-849a-bbf19f441983-kube-api-access-nwtk6\") pod \"network-check-target-2psct\" (UID: \"8a089a4c-8e5a-4878-849a-bbf19f441983\") " pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:47:30.483734 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.483695    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8j5c"
Jul 29 22:47:30.492013 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.491986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-csbtv\""
Jul 29 22:47:30.500696 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.500674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:47:30.631304 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.631220    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8j5c"]
Jul 29 22:47:30.635187 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:47:30.635161    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13db78f9_f4eb_405b_b632_2a9eed835018.slice/crio-a7de1bf427067ceb39cccd93a3eb02fd014d65367570702746bc696102fcba53 WatchSource:0}: Error finding container a7de1bf427067ceb39cccd93a3eb02fd014d65367570702746bc696102fcba53: Status 404 returned error can't find the container with id a7de1bf427067ceb39cccd93a3eb02fd014d65367570702746bc696102fcba53
Jul 29 22:47:30.648306 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.648281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2psct"]
Jul 29 22:47:30.650798 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:47:30.650768    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a089a4c_8e5a_4878_849a_bbf19f441983.slice/crio-314fa01cc05242b5d76f661e0aa109b9817218af1ada3db4662a69d3ad04678e WatchSource:0}: Error finding container 314fa01cc05242b5d76f661e0aa109b9817218af1ada3db4662a69d3ad04678e: Status 404 returned error can't find the container with id 314fa01cc05242b5d76f661e0aa109b9817218af1ada3db4662a69d3ad04678e
Jul 29 22:47:30.823421 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.823340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2psct" event={"ID":"8a089a4c-8e5a-4878-849a-bbf19f441983","Type":"ContainerStarted","Data":"314fa01cc05242b5d76f661e0aa109b9817218af1ada3db4662a69d3ad04678e"}
Jul 29 22:47:30.824398 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:30.824377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8j5c" event={"ID":"13db78f9-f4eb-405b-b632-2a9eed835018","Type":"ContainerStarted","Data":"a7de1bf427067ceb39cccd93a3eb02fd014d65367570702746bc696102fcba53"}
Jul 29 22:47:35.840317 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:35.840278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2psct" event={"ID":"8a089a4c-8e5a-4878-849a-bbf19f441983","Type":"ContainerStarted","Data":"c007ceeb9de9b48c8a1b3816139c89ba1038a44b9df6cea3369e94f6f40c61b6"}
Jul 29 22:47:35.840771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:35.840359    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:47:35.841725 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:35.841700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8j5c" event={"ID":"13db78f9-f4eb-405b-b632-2a9eed835018","Type":"ContainerStarted","Data":"4c528d4e89c5add69284acf5f5ef5aaee7738b29cd13b0e2b8e7ef91036ccad4"}
Jul 29 22:47:35.855079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:35.855037    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2psct" podStartSLOduration=67.459100309 podStartE2EDuration="1m11.855021726s" totalImagesPullingTime="4.395921417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:35.853966285 +0000 UTC m=+71.762656644" watchObservedRunningTime="2026-07-29 22:47:35.855021726 +0000 UTC m=+71.763712137"
Jul 29 22:47:35.866839 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:35.866778    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x8j5c" podStartSLOduration=67.45146047 podStartE2EDuration="1m11.866769498s" totalImagesPullingTime="4.415309028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:35.866497297 +0000 UTC m=+71.775187654" watchObservedRunningTime="2026-07-29 22:47:35.866769498 +0000 UTC m=+71.775459855"
Jul 29 22:47:45.663715 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.663682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"]
Jul 29 22:47:45.665347 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.665331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.667332 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.667311    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 22:47:45.667953 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.667937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 22:47:45.668032 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.667938    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5c79r\""
Jul 29 22:47:45.668032 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.667979    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 22:47:45.668032 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.667997    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 22:47:45.676212 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.676195    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"]
Jul 29 22:47:45.715439 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.715413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2ecedc0-d67e-4dc2-8253-b290f45df95d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86549cbdf8-h4slf\" (UID: \"d2ecedc0-d67e-4dc2-8253-b290f45df95d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.715563 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.715472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2p2\" (UniqueName: \"kubernetes.io/projected/d2ecedc0-d67e-4dc2-8253-b290f45df95d-kube-api-access-9p2p2\") pod \"managed-serviceaccount-addon-agent-86549cbdf8-h4slf\" (UID: \"d2ecedc0-d67e-4dc2-8253-b290f45df95d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.816462 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.816421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2p2\" (UniqueName: \"kubernetes.io/projected/d2ecedc0-d67e-4dc2-8253-b290f45df95d-kube-api-access-9p2p2\") pod \"managed-serviceaccount-addon-agent-86549cbdf8-h4slf\" (UID: \"d2ecedc0-d67e-4dc2-8253-b290f45df95d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.816617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.816490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2ecedc0-d67e-4dc2-8253-b290f45df95d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86549cbdf8-h4slf\" (UID: \"d2ecedc0-d67e-4dc2-8253-b290f45df95d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.819182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.819157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2ecedc0-d67e-4dc2-8253-b290f45df95d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86549cbdf8-h4slf\" (UID: \"d2ecedc0-d67e-4dc2-8253-b290f45df95d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.823630 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.823609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2p2\" (UniqueName: \"kubernetes.io/projected/d2ecedc0-d67e-4dc2-8253-b290f45df95d-kube-api-access-9p2p2\") pod \"managed-serviceaccount-addon-agent-86549cbdf8-h4slf\" (UID: \"d2ecedc0-d67e-4dc2-8253-b290f45df95d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:45.980485 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:45.980370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"
Jul 29 22:47:46.102676 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:46.102646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf"]
Jul 29 22:47:46.105891 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:47:46.105856    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2ecedc0_d67e_4dc2_8253_b290f45df95d.slice/crio-15f0c99ea7a22daf7f7ccc4385b8c0b1b6e2194d5e324e240b2839e6dfb034c8 WatchSource:0}: Error finding container 15f0c99ea7a22daf7f7ccc4385b8c0b1b6e2194d5e324e240b2839e6dfb034c8: Status 404 returned error can't find the container with id 15f0c99ea7a22daf7f7ccc4385b8c0b1b6e2194d5e324e240b2839e6dfb034c8
Jul 29 22:47:46.867833 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:46.867783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf" event={"ID":"d2ecedc0-d67e-4dc2-8253-b290f45df95d","Type":"ContainerStarted","Data":"15f0c99ea7a22daf7f7ccc4385b8c0b1b6e2194d5e324e240b2839e6dfb034c8"}
Jul 29 22:47:48.874290 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:48.874200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf" event={"ID":"d2ecedc0-d67e-4dc2-8253-b290f45df95d","Type":"ContainerStarted","Data":"32937ca2eba837af683d0d63bceacba4eac7e53641f111acc8cf05632e50b45f"}
Jul 29 22:47:48.889546 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:47:48.889499    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf" podStartSLOduration=1.403019961 podStartE2EDuration="3.889484355s" totalImagesPullingTime="2.486464394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:47:48.888503566 +0000 UTC m=+84.797193924" watchObservedRunningTime="2026-07-29 22:47:48.889484355 +0000 UTC m=+84.798174713"
Jul 29 22:48:02.330962 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:48:02.330913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:48:02.331459 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:48:02.331044    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:48:02.331459 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:48:02.331105    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:49:06.331088082 +0000 UTC m=+162.239778422 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:48:06.846690 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:48:06.846664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2psct"
Jul 29 22:48:34.343610 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:48:34.343573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:48:34.344138 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:48:34.343759    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:48:34.344138 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:48:34.343872    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:36.343850338 +0000 UTC m=+252.252540678 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : secret "metrics-daemon-secret" not found
Jul 29 22:49:01.445159 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:49:01.445101    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wt4mt" podUID="7c721bef-22cf-40a8-878c-f071e3a1f012"
Jul 29 22:49:02.045534 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:02.045506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:49:02.595113 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:49:02.595064    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:49:06.349940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:06.349880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:49:06.350564 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:49:06.350088    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 22:49:06.350564 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:49:06.350200    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls podName:7c721bef-22cf-40a8-878c-f071e3a1f012 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:08.350170112 +0000 UTC m=+284.258860465 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls") pod "dns-default-wt4mt" (UID: "7c721bef-22cf-40a8-878c-f071e3a1f012") : secret "dns-default-metrics-tls" not found
Jul 29 22:49:17.564198 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:17.564107    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:49:49.155240 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:49.155201    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2ecedc0-d67e-4dc2-8253-b290f45df95d" containerID="32937ca2eba837af683d0d63bceacba4eac7e53641f111acc8cf05632e50b45f" exitCode=255
Jul 29 22:49:49.155692 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:49.155258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf" event={"ID":"d2ecedc0-d67e-4dc2-8253-b290f45df95d","Type":"ContainerDied","Data":"32937ca2eba837af683d0d63bceacba4eac7e53641f111acc8cf05632e50b45f"}
Jul 29 22:49:49.160824 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:49.160783    2580 scope.go:117] "RemoveContainer" containerID="32937ca2eba837af683d0d63bceacba4eac7e53641f111acc8cf05632e50b45f"
Jul 29 22:49:50.158837 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:49:50.158781    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86549cbdf8-h4slf" event={"ID":"d2ecedc0-d67e-4dc2-8253-b290f45df95d","Type":"ContainerStarted","Data":"db76df55723336799d4f945a54d6d06b29e97a1f4a84d10b100c40c0e4637b89"}
Jul 29 22:50:36.351362 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:36.351320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:50:36.351886 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:36.351470    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 22:50:36.351886 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:36.351545    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs podName:af595ab0-8515-4923-af82-6cdcd20b5c57 nodeName:}" failed. No retries permitted until 2026-07-29 22:52:38.351529684 +0000 UTC m=+374.260220023 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs") pod "network-metrics-daemon-vct9g" (UID: "af595ab0-8515-4923-af82-6cdcd20b5c57") : secret "metrics-daemon-secret" not found
Jul 29 22:50:44.158376 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.158339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"]
Jul 29 22:50:44.161169 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.161148    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"]
Jul 29 22:50:44.161287 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.161262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"
Jul 29 22:50:44.163209 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.163186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wbqmm\""
Jul 29 22:50:44.163781 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.163764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"]
Jul 29 22:50:44.163889 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.163872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.166079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166058    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.166185 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166137    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dd689"]
Jul 29 22:50:44.166250 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166198    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 29 22:50:44.166369 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.166437 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166388    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 29 22:50:44.166437 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166392    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hvrlm\""
Jul 29 22:50:44.166573 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.166436    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.171471 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.171451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 22:50:44.171679 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.171451    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 22:50:44.171759 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.171696    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.171889 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.171870    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qjkv8\""
Jul 29 22:50:44.172005 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.171988    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.172922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.172903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.173719 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.173697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"]
Jul 29 22:50:44.175073 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.175054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 22:50:44.175351 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.175336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jp2v7\""
Jul 29 22:50:44.175491 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.175349    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.175691 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.175676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.175751 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.175704    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 22:50:44.176330 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.176310    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"]
Jul 29 22:50:44.179624 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.179605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"]
Jul 29 22:50:44.181046 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.181029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 22:50:44.193896 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.193873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dd689"]
Jul 29 22:50:44.252826 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.252776    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"]
Jul 29 22:50:44.255801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.255765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:44.257206 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.257187    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d7fb7b5bf-sg26j"]
Jul 29 22:50:44.257722 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.257701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wf28s\""
Jul 29 22:50:44.257966 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.257948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 29 22:50:44.258031 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.258014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.258372 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.258354    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.259890 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.259875    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6765967669-vw2n9"]
Jul 29 22:50:44.260011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.259999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.262083 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.262068    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 22:50:44.262548 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.262532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 29 22:50:44.262705 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.262690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.263479 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.263442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hkpkg\""
Jul 29 22:50:44.263479 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.263468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 29 22:50:44.263620 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.263479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 29 22:50:44.263620 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.263535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 29 22:50:44.263749 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.263735    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 22:50:44.264314 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.264302    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 22:50:44.264371 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.264320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 22:50:44.265188 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.265170    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 22:50:44.267067 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.267052    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vpnhm\""
Jul 29 22:50:44.270689 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.270668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 22:50:44.274034 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.274014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"]
Jul 29 22:50:44.275115 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.275099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d7fb7b5bf-sg26j"]
Jul 29 22:50:44.281526 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.281505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6765967669-vw2n9"]
Jul 29 22:50:44.304599 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-snapshots\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.304744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-config\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.304744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-tmp\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.304744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-service-ca-bundle\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.304744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsmv\" (UniqueName: \"kubernetes.io/projected/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-kube-api-access-jfsmv\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.304744 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-serving-cert\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.305011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dvzb\" (UniqueName: \"kubernetes.io/projected/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-kube-api-access-8dvzb\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.305011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-serving-cert\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.305011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/227b4dbc-e960-4432-b261-f7cf67a14aba-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.305011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.305011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.304981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhvn\" (UniqueName: \"kubernetes.io/projected/32fecf55-924b-4f20-9d95-baef57559c53-kube-api-access-zxhvn\") pod \"network-check-source-7d88cb55d8-pd6zq\" (UID: \"32fecf55-924b-4f20-9d95-baef57559c53\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"
Jul 29 22:50:44.305181 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.305015    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdg47\" (UniqueName: \"kubernetes.io/projected/227b4dbc-e960-4432-b261-f7cf67a14aba-kube-api-access-cdg47\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.305181 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.305037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.405830 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-serving-cert\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.405830 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-default-certificate\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/227b4dbc-e960-4432-b261-f7cf67a14aba-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-installation-pull-secrets\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-bound-sa-token\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7d4\" (UniqueName: \"kubernetes.io/projected/158d70b4-f39d-4677-b046-6ba8118465e1-kube-api-access-hk7d4\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:44.406040 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.405994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5t7\" (UniqueName: \"kubernetes.io/projected/64791bc3-b8dd-4974-bd07-96d0bed3c998-kube-api-access-2b5t7\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.406313 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hnf\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-kube-api-access-95hnf\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-image-registry-private-configuration\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdg47\" (UniqueName: \"kubernetes.io/projected/227b4dbc-e960-4432-b261-f7cf67a14aba-kube-api-access-cdg47\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-registry-certificates\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-snapshots\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-tmp\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-serving-cert\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dvzb\" (UniqueName: \"kubernetes.io/projected/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-kube-api-access-8dvzb\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-service-ca-bundle\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsmv\" (UniqueName: \"kubernetes.io/projected/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-kube-api-access-jfsmv\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-stats-auth\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.406976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-config\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/227b4dbc-e960-4432-b261-f7cf67a14aba-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.411260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-snapshots\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68d8774d-2013-49df-9eb5-a0052657dc30-ca-trust-extracted\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhvn\" (UniqueName: \"kubernetes.io/projected/32fecf55-924b-4f20-9d95-baef57559c53-kube-api-access-zxhvn\") pod \"network-check-source-7d88cb55d8-pd6zq\" (UID: \"32fecf55-924b-4f20-9d95-baef57559c53\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-config\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-trusted-ca\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407542    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-tmp\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.407639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.407948    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.408012    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls podName:227b4dbc-e960-4432-b261-f7cf67a14aba nodeName:}" failed. No retries permitted until 2026-07-29 22:50:44.907993614 +0000 UTC m=+260.816683952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tg52h" (UID: "227b4dbc-e960-4432-b261-f7cf67a14aba") : secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.408371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-service-ca-bundle\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.411873 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.408884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-serving-cert\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.413246 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.413227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-serving-cert\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.415984 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.415955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhvn\" (UniqueName: \"kubernetes.io/projected/32fecf55-924b-4f20-9d95-baef57559c53-kube-api-access-zxhvn\") pod \"network-check-source-7d88cb55d8-pd6zq\" (UID: \"32fecf55-924b-4f20-9d95-baef57559c53\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"
Jul 29 22:50:44.416173 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.416154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dvzb\" (UniqueName: \"kubernetes.io/projected/93aa25fa-17e8-49c4-aad3-2fcc8d472c73-kube-api-access-8dvzb\") pod \"service-ca-operator-5d6c75f84f-htbcm\" (UID: \"93aa25fa-17e8-49c4-aad3-2fcc8d472c73\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.416929 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.416906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdg47\" (UniqueName: \"kubernetes.io/projected/227b4dbc-e960-4432-b261-f7cf67a14aba-kube-api-access-cdg47\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.417025 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.416929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsmv\" (UniqueName: \"kubernetes.io/projected/258f5c2b-06f6-4659-a589-afdbe1c6bf5e-kube-api-access-jfsmv\") pod \"insights-operator-5885fdb759-dd689\" (UID: \"258f5c2b-06f6-4659-a589-afdbe1c6bf5e\") " pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.473922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.473887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"
Jul 29 22:50:44.481397 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.481375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"
Jul 29 22:50:44.492342 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.492318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-dd689"
Jul 29 22:50:44.508196 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.508331 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-default-certificate\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.508404 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.508361    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:45.008336728 +0000 UTC m=+260.917027080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : configmap references non-existent config key: service-ca.crt
Jul 29 22:50:44.508476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-installation-pull-secrets\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.508476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-bound-sa-token\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.508592 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7d4\" (UniqueName: \"kubernetes.io/projected/158d70b4-f39d-4677-b046-6ba8118465e1-kube-api-access-hk7d4\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:44.508592 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5t7\" (UniqueName: \"kubernetes.io/projected/64791bc3-b8dd-4974-bd07-96d0bed3c998-kube-api-access-2b5t7\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.508592 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hnf\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-kube-api-access-95hnf\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.508592 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-image-registry-private-configuration\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.508802 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-registry-certificates\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.508802 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:44.508802 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.508802 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-stats-auth\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.508802 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.509100 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68d8774d-2013-49df-9eb5-a0052657dc30-ca-trust-extracted\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.509100 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.508851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-trusted-ca\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.509286 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509264    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 22:50:44.509357 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509344    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:45.009324715 +0000 UTC m=+260.918015056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : secret "router-metrics-certs-default" not found
Jul 29 22:50:44.509419 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.509375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-registry-certificates\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.509666 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509645    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 22:50:44.509751 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509696    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls podName:158d70b4-f39d-4677-b046-6ba8118465e1 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:45.009682853 +0000 UTC m=+260.918373208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls") pod "cluster-samples-operator-6748489656-x97fb" (UID: "158d70b4-f39d-4677-b046-6ba8118465e1") : secret "samples-operator-tls" not found
Jul 29 22:50:44.509833 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.509745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-trusted-ca\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.509895 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509861    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:50:44.509895 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509877    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6765967669-vw2n9: secret "image-registry-tls" not found
Jul 29 22:50:44.510000 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.509919    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls podName:68d8774d-2013-49df-9eb5-a0052657dc30 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:45.009904178 +0000 UTC m=+260.918594515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls") pod "image-registry-6765967669-vw2n9" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30") : secret "image-registry-tls" not found
Jul 29 22:50:44.510416 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.510392    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68d8774d-2013-49df-9eb5-a0052657dc30-ca-trust-extracted\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.512326 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.512299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-installation-pull-secrets\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.512494 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.512473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-stats-auth\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.512814 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.512782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-default-certificate\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.512932 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.512910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-image-registry-private-configuration\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.518158 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.518125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5t7\" (UniqueName: \"kubernetes.io/projected/64791bc3-b8dd-4974-bd07-96d0bed3c998-kube-api-access-2b5t7\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:44.518703 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.518682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7d4\" (UniqueName: \"kubernetes.io/projected/158d70b4-f39d-4677-b046-6ba8118465e1-kube-api-access-hk7d4\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:44.519214 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.519187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hnf\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-kube-api-access-95hnf\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.520138 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.520112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-bound-sa-token\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:44.616120 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.616070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq"]
Jul 29 22:50:44.619638 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:50:44.619611    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fecf55_924b_4f20_9d95_baef57559c53.slice/crio-0f630ad841ebdfdc77ea16b793cb33037c0a8a7a80c13c387c0cf7b9f6966e63 WatchSource:0}: Error finding container 0f630ad841ebdfdc77ea16b793cb33037c0a8a7a80c13c387c0cf7b9f6966e63: Status 404 returned error can't find the container with id 0f630ad841ebdfdc77ea16b793cb33037c0a8a7a80c13c387c0cf7b9f6966e63
Jul 29 22:50:44.635042 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.635015    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm"]
Jul 29 22:50:44.638322 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:50:44.638285    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aa25fa_17e8_49c4_aad3_2fcc8d472c73.slice/crio-c19107c698ea9a1583d5fab57fb6fc6a51ee0f8b15994d00ec6edf59828f1195 WatchSource:0}: Error finding container c19107c698ea9a1583d5fab57fb6fc6a51ee0f8b15994d00ec6edf59828f1195: Status 404 returned error can't find the container with id c19107c698ea9a1583d5fab57fb6fc6a51ee0f8b15994d00ec6edf59828f1195
Jul 29 22:50:44.654355 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.654331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-dd689"]
Jul 29 22:50:44.669729 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:50:44.669706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258f5c2b_06f6_4659_a589_afdbe1c6bf5e.slice/crio-ee834ba2bf1d42a0b12ff0a24157dafcbc6058565e79f2aac254df09d1e8d6ab WatchSource:0}: Error finding container ee834ba2bf1d42a0b12ff0a24157dafcbc6058565e79f2aac254df09d1e8d6ab: Status 404 returned error can't find the container with id ee834ba2bf1d42a0b12ff0a24157dafcbc6058565e79f2aac254df09d1e8d6ab
Jul 29 22:50:44.912372 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:44.912335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:44.912575 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.912455    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:44.912575 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:44.912510    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls podName:227b4dbc-e960-4432-b261-f7cf67a14aba nodeName:}" failed. No retries permitted until 2026-07-29 22:50:45.912495323 +0000 UTC m=+261.821185663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tg52h" (UID: "227b4dbc-e960-4432-b261-f7cf67a14aba") : secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:45.013188 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.013095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:45.013188 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.013147    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:45.013414 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.013254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:45.013414 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013324    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:50:45.013414 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013338    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6765967669-vw2n9: secret "image-registry-tls" not found
Jul 29 22:50:45.013414 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.013334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:45.013414 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013392    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls podName:68d8774d-2013-49df-9eb5-a0052657dc30 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:46.013373801 +0000 UTC m=+261.922064141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls") pod "image-registry-6765967669-vw2n9" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30") : secret "image-registry-tls" not found
Jul 29 22:50:45.013621 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013417    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 22:50:45.013621 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:46.013420097 +0000 UTC m=+261.922110440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : configmap references non-existent config key: service-ca.crt
Jul 29 22:50:45.013621 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013464    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls podName:158d70b4-f39d-4677-b046-6ba8118465e1 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:46.013450472 +0000 UTC m=+261.922140826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls") pod "cluster-samples-operator-6748489656-x97fb" (UID: "158d70b4-f39d-4677-b046-6ba8118465e1") : secret "samples-operator-tls" not found
Jul 29 22:50:45.013621 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013420    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 22:50:45.013621 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.013499    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:46.013489213 +0000 UTC m=+261.922179550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : secret "router-metrics-certs-default" not found
Jul 29 22:50:45.292591 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.292512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dd689" event={"ID":"258f5c2b-06f6-4659-a589-afdbe1c6bf5e","Type":"ContainerStarted","Data":"ee834ba2bf1d42a0b12ff0a24157dafcbc6058565e79f2aac254df09d1e8d6ab"}
Jul 29 22:50:45.293777 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.293743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm" event={"ID":"93aa25fa-17e8-49c4-aad3-2fcc8d472c73","Type":"ContainerStarted","Data":"c19107c698ea9a1583d5fab57fb6fc6a51ee0f8b15994d00ec6edf59828f1195"}
Jul 29 22:50:45.295334 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.295310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq" event={"ID":"32fecf55-924b-4f20-9d95-baef57559c53","Type":"ContainerStarted","Data":"bd6910571b778762e11c04623cd2b08fadc87f1157e4169a72785a532444c96e"}
Jul 29 22:50:45.295466 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.295351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq" event={"ID":"32fecf55-924b-4f20-9d95-baef57559c53","Type":"ContainerStarted","Data":"0f630ad841ebdfdc77ea16b793cb33037c0a8a7a80c13c387c0cf7b9f6966e63"}
Jul 29 22:50:45.310708 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.310658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pd6zq" podStartSLOduration=1.3106401650000001 podStartE2EDuration="1.310640165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:45.309825521 +0000 UTC m=+261.218515880" watchObservedRunningTime="2026-07-29 22:50:45.310640165 +0000 UTC m=+261.219330529"
Jul 29 22:50:45.921117 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:45.921069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:45.921302 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.921242    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:45.921363 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:45.921327    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls podName:227b4dbc-e960-4432-b261-f7cf67a14aba nodeName:}" failed. No retries permitted until 2026-07-29 22:50:47.92130455 +0000 UTC m=+263.829994899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tg52h" (UID: "227b4dbc-e960-4432-b261-f7cf67a14aba") : secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:46.022111 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:46.022068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:46.022306 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:46.022127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:46.022306 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:46.022165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:46.022306 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:46.022240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:46.022306 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022249    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 22:50:46.022306 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022294    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:50:46.022570 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022314    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6765967669-vw2n9: secret "image-registry-tls" not found
Jul 29 22:50:46.022570 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022323    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls podName:158d70b4-f39d-4677-b046-6ba8118465e1 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:48.022303027 +0000 UTC m=+263.930993378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls") pod "cluster-samples-operator-6748489656-x97fb" (UID: "158d70b4-f39d-4677-b046-6ba8118465e1") : secret "samples-operator-tls" not found
Jul 29 22:50:46.022570 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022366    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls podName:68d8774d-2013-49df-9eb5-a0052657dc30 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:48.022348574 +0000 UTC m=+263.931038934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls") pod "image-registry-6765967669-vw2n9" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30") : secret "image-registry-tls" not found
Jul 29 22:50:46.022570 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022371    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 22:50:46.022570 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022385    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:48.022375578 +0000 UTC m=+263.931065914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : configmap references non-existent config key: service-ca.crt
Jul 29 22:50:46.022570 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:46.022425    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:48.02241045 +0000 UTC m=+263.931100790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : secret "router-metrics-certs-default" not found
Jul 29 22:50:47.304818 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:47.304760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm" event={"ID":"93aa25fa-17e8-49c4-aad3-2fcc8d472c73","Type":"ContainerStarted","Data":"9a38a766439ca2eb3f8542c5e376d4a4dcbf15285430be31f9d63befc54e6b79"}
Jul 29 22:50:47.306371 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:47.306346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dd689" event={"ID":"258f5c2b-06f6-4659-a589-afdbe1c6bf5e","Type":"ContainerStarted","Data":"6a33de7039440c77a61e8a2fcc26c6a181136d3dcc06c5bd7503060b50d1e726"}
Jul 29 22:50:47.318405 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:47.318359    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm" podStartSLOduration=0.769915239 podStartE2EDuration="3.31834535s" totalImagesPullingTime="2.548430111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:47.317679105 +0000 UTC m=+263.226369464" watchObservedRunningTime="2026-07-29 22:50:47.31834535 +0000 UTC m=+263.227035707"
Jul 29 22:50:47.332261 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:47.332205    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-dd689" podStartSLOduration=0.818136231 podStartE2EDuration="3.332185514s" totalImagesPullingTime="2.514049283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:50:47.33173346 +0000 UTC m=+263.240423819" watchObservedRunningTime="2026-07-29 22:50:47.332185514 +0000 UTC m=+263.240875875"
Jul 29 22:50:47.938055 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:47.938011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:47.938235 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:47.938210    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:47.938309 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:47.938298    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls podName:227b4dbc-e960-4432-b261-f7cf67a14aba nodeName:}" failed. No retries permitted until 2026-07-29 22:50:51.93827633 +0000 UTC m=+267.846966671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tg52h" (UID: "227b4dbc-e960-4432-b261-f7cf67a14aba") : secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:48.038653 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.038599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:48.038848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.038667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:48.038848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.038733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:48.038848 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038805    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:50:48.038848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.038819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038835    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6765967669-vw2n9: secret "image-registry-tls" not found
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038894    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038817    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.038895843 +0000 UTC m=+267.947586181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : configmap references non-existent config key: service-ca.crt
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls podName:68d8774d-2013-49df-9eb5-a0052657dc30 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.038935256 +0000 UTC m=+267.947625597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls") pod "image-registry-6765967669-vw2n9" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30") : secret "image-registry-tls" not found
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038962    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.038952675 +0000 UTC m=+267.947643054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : secret "router-metrics-certs-default" not found
Jul 29 22:50:48.039042 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.038976    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls podName:158d70b4-f39d-4677-b046-6ba8118465e1 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.038968294 +0000 UTC m=+267.947658636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls") pod "cluster-samples-operator-6748489656-x97fb" (UID: "158d70b4-f39d-4677-b046-6ba8118465e1") : secret "samples-operator-tls" not found
Jul 29 22:50:48.737689 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.737653    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8rxsh"]
Jul 29 22:50:48.740979 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.740959    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.742882 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.742861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 22:50:48.742976 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.742891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfc6t\""
Jul 29 22:50:48.743376 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.743354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 22:50:48.750783 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.750762    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8rxsh"]
Jul 29 22:50:48.844695 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.844661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0775c5fc-3a94-46dd-8691-b29c2db1e186-data-volume\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.844695 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.844697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0775c5fc-3a94-46dd-8691-b29c2db1e186-crio-socket\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.844908 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.844721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclk9\" (UniqueName: \"kubernetes.io/projected/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-api-access-sclk9\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.844908 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.844818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.844977 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.844910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945400 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945539 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945539 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0775c5fc-3a94-46dd-8691-b29c2db1e186-data-volume\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945539 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0775c5fc-3a94-46dd-8691-b29c2db1e186-crio-socket\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945539 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclk9\" (UniqueName: \"kubernetes.io/projected/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-api-access-sclk9\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945695 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.945510    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:48.945695 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.945583    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:48.945695 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0775c5fc-3a94-46dd-8691-b29c2db1e186-crio-socket\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.945695 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.945644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:49.445624267 +0000 UTC m=+265.354314617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:48.945695 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:48.945669    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:49.445658754 +0000 UTC m=+265.354349093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:48.945907 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.945889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0775c5fc-3a94-46dd-8691-b29c2db1e186-data-volume\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:48.955326 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:48.955304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclk9\" (UniqueName: \"kubernetes.io/projected/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-api-access-sclk9\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:49.449133 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:49.449098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:49.449318 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:49.449165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:49.449318 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:49.449243    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:49.449318 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:49.449273    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:49.449318 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:49.449310    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:50.449289039 +0000 UTC m=+266.357979399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:49.449454 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:49.449326    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:50.449318546 +0000 UTC m=+266.358008883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:50.457549 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:50.457512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:50.458024 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:50.457652    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:50.458024 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:50.457715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:50.458024 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:50.457733    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.45771234 +0000 UTC m=+268.366402676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:50.458024 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:50.457813    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:50.458024 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:50.457857    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:52.457844235 +0000 UTC m=+268.366534574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:51.541634 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:51.541608    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ts59j_df0d9576-7662-40f9-b242-50eb3acada25/dns-node-resolver/0.log"
Jul 29 22:50:51.973027 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:51.972986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:50:51.973229 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:51.973191    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:51.973295 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:51.973277    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls podName:227b4dbc-e960-4432-b261-f7cf67a14aba nodeName:}" failed. No retries permitted until 2026-07-29 22:50:59.973254741 +0000 UTC m=+275.881945077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tg52h" (UID: "227b4dbc-e960-4432-b261-f7cf67a14aba") : secret "cluster-monitoring-operator-tls" not found
Jul 29 22:50:52.074007 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.073972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:52.074193 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.074150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:50:52.074193 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074156    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:00.074138662 +0000 UTC m=+275.982828999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : configmap references non-existent config key: service-ca.crt
Jul 29 22:50:52.074282 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.074201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:50:52.074282 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.074247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:50:52.074349 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074284    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 29 22:50:52.074349 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074301    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 22:50:52.074349 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074318    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6765967669-vw2n9: secret "image-registry-tls" not found
Jul 29 22:50:52.074349 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074333    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls podName:158d70b4-f39d-4677-b046-6ba8118465e1 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:00.074317871 +0000 UTC m=+275.983008207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls") pod "cluster-samples-operator-6748489656-x97fb" (UID: "158d70b4-f39d-4677-b046-6ba8118465e1") : secret "samples-operator-tls" not found
Jul 29 22:50:52.074477 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074353    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls podName:68d8774d-2013-49df-9eb5-a0052657dc30 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:00.074342556 +0000 UTC m=+275.983032895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls") pod "image-registry-6765967669-vw2n9" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30") : secret "image-registry-tls" not found
Jul 29 22:50:52.074477 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074401    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 29 22:50:52.074477 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.074457    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:00.074442724 +0000 UTC m=+275.983133064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : secret "router-metrics-certs-default" not found
Jul 29 22:50:52.478146 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.478106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:52.478337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.478176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:52.478337 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.478247    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:52.478337 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.478293    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:52.478337 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.478315    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:56.478298968 +0000 UTC m=+272.386989303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:52.478519 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:52.478358    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:50:56.478338488 +0000 UTC m=+272.387028838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:52.541493 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:52.541464    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hg8hl_062d05c7-b02b-464f-abb6-3c6f113ee0ce/node-ca/0.log"
Jul 29 22:50:53.142186 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:53.142132    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4hpqx_00a5e172-5fa6-4f97-b08b-33d512358027/serve-healthcheck-canary/0.log"
Jul 29 22:50:56.511260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:56.511225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:56.511615 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:50:56.511284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:50:56.511615 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:56.511345    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 22:50:56.511615 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:56.511369    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:50:56.511615 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:56.511399    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:04.511385864 +0000 UTC m=+280.420076203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : configmap "kube-rbac-proxy" not found
Jul 29 22:50:56.511615 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:50:56.511420    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:04.511407038 +0000 UTC m=+280.420097378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:00.040742 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.040702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:51:00.041228 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:00.040844    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 22:51:00.041228 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:00.040914    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls podName:227b4dbc-e960-4432-b261-f7cf67a14aba nodeName:}" failed. No retries permitted until 2026-07-29 22:51:16.040892359 +0000 UTC m=+291.949582696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-tg52h" (UID: "227b4dbc-e960-4432-b261-f7cf67a14aba") : secret "cluster-monitoring-operator-tls" not found
Jul 29 22:51:00.141618 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.141583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:51:00.141831 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.141637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:00.141831 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.141672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:00.141831 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.141748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:00.141997 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:00.141919    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle podName:64791bc3-b8dd-4974-bd07-96d0bed3c998 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:16.141899452 +0000 UTC m=+292.050589805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle") pod "router-default-6d7fb7b5bf-sg26j" (UID: "64791bc3-b8dd-4974-bd07-96d0bed3c998") : configmap references non-existent config key: service-ca.crt
Jul 29 22:51:00.144109 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.144082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64791bc3-b8dd-4974-bd07-96d0bed3c998-metrics-certs\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:00.144211 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.144197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/158d70b4-f39d-4677-b046-6ba8118465e1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-x97fb\" (UID: \"158d70b4-f39d-4677-b046-6ba8118465e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:51:00.144249 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.144218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"image-registry-6765967669-vw2n9\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") " pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:00.167031 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.167008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"
Jul 29 22:51:00.180175 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.180156    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:00.293806 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.293714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb"]
Jul 29 22:51:00.313197 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.313171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6765967669-vw2n9"]
Jul 29 22:51:00.316072 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:00.316044    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d8774d_2013_49df_9eb5_a0052657dc30.slice/crio-8b222aa41d2ed43e7ebce0da46620ee30118ff8c66bda79a075f5d0ac84bbde0 WatchSource:0}: Error finding container 8b222aa41d2ed43e7ebce0da46620ee30118ff8c66bda79a075f5d0ac84bbde0: Status 404 returned error can't find the container with id 8b222aa41d2ed43e7ebce0da46620ee30118ff8c66bda79a075f5d0ac84bbde0
Jul 29 22:51:00.347811 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.347762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6765967669-vw2n9" event={"ID":"68d8774d-2013-49df-9eb5-a0052657dc30","Type":"ContainerStarted","Data":"8b222aa41d2ed43e7ebce0da46620ee30118ff8c66bda79a075f5d0ac84bbde0"}
Jul 29 22:51:00.348874 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:00.348852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb" event={"ID":"158d70b4-f39d-4677-b046-6ba8118465e1","Type":"ContainerStarted","Data":"9a5822bced0e3145162af98992a38d1f7d1253b71b44119ad5dbb8026397d689"}
Jul 29 22:51:01.353857 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:01.353824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6765967669-vw2n9" event={"ID":"68d8774d-2013-49df-9eb5-a0052657dc30","Type":"ContainerStarted","Data":"cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca"}
Jul 29 22:51:01.354296 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:01.353953    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:01.377415 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:01.377359    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6765967669-vw2n9" podStartSLOduration=17.377340066 podStartE2EDuration="17.377340066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:01.376112718 +0000 UTC m=+277.284803077" watchObservedRunningTime="2026-07-29 22:51:01.377340066 +0000 UTC m=+277.286030425"
Jul 29 22:51:02.358565 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:02.358531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb" event={"ID":"158d70b4-f39d-4677-b046-6ba8118465e1","Type":"ContainerStarted","Data":"e95cc49405e8c22f9d4dd4edb1a7324d1ae19959cb789475b4ecaea23304af70"}
Jul 29 22:51:02.359054 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:02.358628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb" event={"ID":"158d70b4-f39d-4677-b046-6ba8118465e1","Type":"ContainerStarted","Data":"00b8722c80b75890493c79a3588ca3ec494dbae2dd6b3d77d708cd7ea0ac2cfb"}
Jul 29 22:51:02.375778 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:02.375724    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-x97fb" podStartSLOduration=16.582750296 podStartE2EDuration="18.375706715s" totalImagesPullingTime="1.792956419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:02.374983615 +0000 UTC m=+278.283673991" watchObservedRunningTime="2026-07-29 22:51:02.375706715 +0000 UTC m=+278.284397073"
Jul 29 22:51:04.579698 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:04.579665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:51:04.580119 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:04.579770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:51:04.580119 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:04.579904    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.580119 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:04.579977    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls podName:0775c5fc-3a94-46dd-8691-b29c2db1e186 nodeName:}" failed. No retries permitted until 2026-07-29 22:51:20.579957093 +0000 UTC m=+296.488647442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rxsh" (UID: "0775c5fc-3a94-46dd-8691-b29c2db1e186") : secret "insights-runtime-extractor-tls" not found
Jul 29 22:51:04.580389 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:04.580368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0775c5fc-3a94-46dd-8691-b29c2db1e186-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:51:05.046006 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:05.045925    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wt4mt" podUID="7c721bef-22cf-40a8-878c-f071e3a1f012"
Jul 29 22:51:05.368894 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:05.368866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:51:08.408843 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:08.408805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:51:08.411160 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:08.411135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c721bef-22cf-40a8-878c-f071e3a1f012-metrics-tls\") pod \"dns-default-wt4mt\" (UID: \"7c721bef-22cf-40a8-878c-f071e3a1f012\") " pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:51:08.671822 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:08.671720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jtttq\""
Jul 29 22:51:08.680438 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:08.680407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:51:08.809031 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:08.809000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wt4mt"]
Jul 29 22:51:08.811936 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:08.811905    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c721bef_22cf_40a8_878c_f071e3a1f012.slice/crio-f98aab07e248a512bd44341d0a28fb4bd9c7dfd717da3c504adf9e9e64e76dcb WatchSource:0}: Error finding container f98aab07e248a512bd44341d0a28fb4bd9c7dfd717da3c504adf9e9e64e76dcb: Status 404 returned error can't find the container with id f98aab07e248a512bd44341d0a28fb4bd9c7dfd717da3c504adf9e9e64e76dcb
Jul 29 22:51:09.382680 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:09.382646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt4mt" event={"ID":"7c721bef-22cf-40a8-878c-f071e3a1f012","Type":"ContainerStarted","Data":"f98aab07e248a512bd44341d0a28fb4bd9c7dfd717da3c504adf9e9e64e76dcb"}
Jul 29 22:51:11.390861 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:11.390828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt4mt" event={"ID":"7c721bef-22cf-40a8-878c-f071e3a1f012","Type":"ContainerStarted","Data":"62f9d5caeb92aa0aebba4b85f81493c27ce929bc398bc942e6930e3ed65e5d31"}
Jul 29 22:51:11.391317 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:11.390879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wt4mt" event={"ID":"7c721bef-22cf-40a8-878c-f071e3a1f012","Type":"ContainerStarted","Data":"b9d17cc02d65427d303745959a08b4edfbe2896a2ccacd62252405b1717fd4e4"}
Jul 29 22:51:11.391317 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:11.390975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:51:11.408309 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:11.408244    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wt4mt" podStartSLOduration=251.770654952 podStartE2EDuration="4m13.408228943s" totalImagesPullingTime="1.637573991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:11.407603144 +0000 UTC m=+287.316293503" watchObservedRunningTime="2026-07-29 22:51:11.408228943 +0000 UTC m=+287.316919278"
Jul 29 22:51:12.963670 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:12.963633    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6765967669-vw2n9"]
Jul 29 22:51:16.068250 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.068209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:51:16.070915 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.070892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/227b4dbc-e960-4432-b261-f7cf67a14aba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-tg52h\" (UID: \"227b4dbc-e960-4432-b261-f7cf67a14aba\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:51:16.169202 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.169165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:16.169719 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.169700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64791bc3-b8dd-4974-bd07-96d0bed3c998-service-ca-bundle\") pod \"router-default-6d7fb7b5bf-sg26j\" (UID: \"64791bc3-b8dd-4974-bd07-96d0bed3c998\") " pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:16.287491 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.287449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"
Jul 29 22:51:16.375414 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.375385    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:16.415257 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.415215    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h"]
Jul 29 22:51:16.418271 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:16.418225    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227b4dbc_e960_4432_b261_f7cf67a14aba.slice/crio-8b5570ae7360a947411167f2a30a709abf3e33d5da54659f501f3035411c9e68 WatchSource:0}: Error finding container 8b5570ae7360a947411167f2a30a709abf3e33d5da54659f501f3035411c9e68: Status 404 returned error can't find the container with id 8b5570ae7360a947411167f2a30a709abf3e33d5da54659f501f3035411c9e68
Jul 29 22:51:16.516059 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:16.516025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d7fb7b5bf-sg26j"]
Jul 29 22:51:16.519965 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:16.519930    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64791bc3_b8dd_4974_bd07_96d0bed3c998.slice/crio-fbccc1e1193f128be2a4df4f5c41f0ae9ce769cddaaf68c4b8c178d7ad1c741a WatchSource:0}: Error finding container fbccc1e1193f128be2a4df4f5c41f0ae9ce769cddaaf68c4b8c178d7ad1c741a: Status 404 returned error can't find the container with id fbccc1e1193f128be2a4df4f5c41f0ae9ce769cddaaf68c4b8c178d7ad1c741a
Jul 29 22:51:17.412480 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:17.412442    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j" event={"ID":"64791bc3-b8dd-4974-bd07-96d0bed3c998","Type":"ContainerStarted","Data":"0f63b1a0c643f1825c49a761f6f7a8c39e5c41f08177cda53d5a86717244595b"}
Jul 29 22:51:17.412953 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:17.412506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j" event={"ID":"64791bc3-b8dd-4974-bd07-96d0bed3c998","Type":"ContainerStarted","Data":"fbccc1e1193f128be2a4df4f5c41f0ae9ce769cddaaf68c4b8c178d7ad1c741a"}
Jul 29 22:51:17.413845 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:17.413820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h" event={"ID":"227b4dbc-e960-4432-b261-f7cf67a14aba","Type":"ContainerStarted","Data":"8b5570ae7360a947411167f2a30a709abf3e33d5da54659f501f3035411c9e68"}
Jul 29 22:51:17.430556 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:17.430507    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j" podStartSLOduration=33.43049197 podStartE2EDuration="33.43049197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:17.429314986 +0000 UTC m=+293.338005345" watchObservedRunningTime="2026-07-29 22:51:17.43049197 +0000 UTC m=+293.339182328"
Jul 29 22:51:18.375683 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:18.375643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:18.378358 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:18.378335    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:18.419004 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:18.418965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h" event={"ID":"227b4dbc-e960-4432-b261-f7cf67a14aba","Type":"ContainerStarted","Data":"2f74b6271e6627331054748cec6520c01bccef4c5e7d84fb34714f9e9762eadc"}
Jul 29 22:51:18.419449 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:18.419211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:18.420589 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:18.420568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d7fb7b5bf-sg26j"
Jul 29 22:51:18.433627 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:18.433579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-tg52h" podStartSLOduration=32.523166657 podStartE2EDuration="34.433563658s" totalImagesPullingTime="1.910397001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:18.432767821 +0000 UTC m=+294.341458180" watchObservedRunningTime="2026-07-29 22:51:18.433563658 +0000 UTC m=+294.342254015"
Jul 29 22:51:20.564713 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:20.564668    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vct9g" podUID="af595ab0-8515-4923-af82-6cdcd20b5c57"
Jul 29 22:51:20.599594 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:20.599553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:51:20.601771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:20.601743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0775c5fc-3a94-46dd-8691-b29c2db1e186-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rxsh\" (UID: \"0775c5fc-3a94-46dd-8691-b29c2db1e186\") " pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:51:20.850476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:20.850383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8rxsh"
Jul 29 22:51:20.972452 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:20.972416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8rxsh"]
Jul 29 22:51:20.975151 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:20.975120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0775c5fc_3a94_46dd_8691_b29c2db1e186.slice/crio-f866a2486a87ee4b1dbe6ba438c614de137eaf54604928a2cdfa059c455e2ed4 WatchSource:0}: Error finding container f866a2486a87ee4b1dbe6ba438c614de137eaf54604928a2cdfa059c455e2ed4: Status 404 returned error can't find the container with id f866a2486a87ee4b1dbe6ba438c614de137eaf54604928a2cdfa059c455e2ed4
Jul 29 22:51:21.395871 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:21.395839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wt4mt"
Jul 29 22:51:21.434161 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:21.434125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rxsh" event={"ID":"0775c5fc-3a94-46dd-8691-b29c2db1e186","Type":"ContainerStarted","Data":"f866a2486a87ee4b1dbe6ba438c614de137eaf54604928a2cdfa059c455e2ed4"}
Jul 29 22:51:22.970025 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:22.969995    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:25.679102 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:25.679077    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 22:51:25.748953 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:25.748936    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:51:26.456942 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:26.456908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rxsh" event={"ID":"0775c5fc-3a94-46dd-8691-b29c2db1e186","Type":"ContainerStarted","Data":"a95a805834e4b02f02ea6f7cd55da83f8fc1efbc76fd62859f993b3274d15d07"}
Jul 29 22:51:29.468617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:29.468582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rxsh" event={"ID":"0775c5fc-3a94-46dd-8691-b29c2db1e186","Type":"ContainerStarted","Data":"a1d19372303e3e88fe14165c5a1268d5d6e5e45f3f29951ee0498e23ed99d75c"}
Jul 29 22:51:31.561982 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.561949    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4zb96"]
Jul 29 22:51:31.568602 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.568559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.572261 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.572029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 22:51:31.572261 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.572058    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 22:51:31.573476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.572289    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxcw6\""
Jul 29 22:51:31.573476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.573103    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 22:51:31.573476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.573323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 22:51:31.587807 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.587762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.587937 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.587812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-sys\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.587937 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.587840    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cece57cc-fa61-43a5-b75a-cfb4e4285af7-metrics-client-ca\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.587937 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.587883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.588123 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.587997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-textfile\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.588123 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.588023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-tls\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.588123 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.588051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-root\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.588123 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.588093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7xn\" (UniqueName: \"kubernetes.io/projected/cece57cc-fa61-43a5-b75a-cfb4e4285af7-kube-api-access-7t7xn\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.588268 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.588143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-wtmp\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689409 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-wtmp\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689585 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689585 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-sys\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689585 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cece57cc-fa61-43a5-b75a-cfb4e4285af7-metrics-client-ca\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689585 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689585 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-sys\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689585 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-wtmp\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-textfile\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-tls\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-root\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7xn\" (UniqueName: \"kubernetes.io/projected/cece57cc-fa61-43a5-b75a-cfb4e4285af7-kube-api-access-7t7xn\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cece57cc-fa61-43a5-b75a-cfb4e4285af7-root\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.689940 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.689914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-textfile\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.690478 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.690432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.690578 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.690481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cece57cc-fa61-43a5-b75a-cfb4e4285af7-metrics-client-ca\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.692328 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.692304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.692476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.692453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cece57cc-fa61-43a5-b75a-cfb4e4285af7-node-exporter-tls\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.698333 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.698300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7xn\" (UniqueName: \"kubernetes.io/projected/cece57cc-fa61-43a5-b75a-cfb4e4285af7-kube-api-access-7t7xn\") pod \"node-exporter-4zb96\" (UID: \"cece57cc-fa61-43a5-b75a-cfb4e4285af7\") " pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:31.881289 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:31.881212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4zb96"
Jul 29 22:51:32.563324 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:32.563287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:51:32.834324 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:32.834286    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcece57cc_fa61_43a5_b75a_cfb4e4285af7.slice/crio-29ead1dac53e7e8b85ee1ec15b1c5db68dd7abd1f3fab22aae4e9ba6cfe48cb2 WatchSource:0}: Error finding container 29ead1dac53e7e8b85ee1ec15b1c5db68dd7abd1f3fab22aae4e9ba6cfe48cb2: Status 404 returned error can't find the container with id 29ead1dac53e7e8b85ee1ec15b1c5db68dd7abd1f3fab22aae4e9ba6cfe48cb2
Jul 29 22:51:33.485211 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:33.485168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rxsh" event={"ID":"0775c5fc-3a94-46dd-8691-b29c2db1e186","Type":"ContainerStarted","Data":"d19f6256f9010a1d6c3358469fae138221a349c00e86482fc508a6fcc7e181d8"}
Jul 29 22:51:33.486378 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:33.486351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zb96" event={"ID":"cece57cc-fa61-43a5-b75a-cfb4e4285af7","Type":"ContainerStarted","Data":"29ead1dac53e7e8b85ee1ec15b1c5db68dd7abd1f3fab22aae4e9ba6cfe48cb2"}
Jul 29 22:51:33.505442 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:33.505382    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8rxsh" podStartSLOduration=33.708442599 podStartE2EDuration="45.505362198s" totalImagesPullingTime="11.796919599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:50:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:33.504371088 +0000 UTC m=+309.413061482" watchObservedRunningTime="2026-07-29 22:51:33.505362198 +0000 UTC m=+309.414052558"
Jul 29 22:51:34.411022 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.410988    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b66997cdf-r8knr"]
Jul 29 22:51:34.413553 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.413537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.416338 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.416314    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 22:51:34.416498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.416342    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 22:51:34.416624 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.416611    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 22:51:34.416838 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.416819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7k8qlj8qq0v5f\""
Jul 29 22:51:34.417883 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.417869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 22:51:34.417883 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.417876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 22:51:34.418038 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.417877    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8fcfv\""
Jul 29 22:51:34.430882 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.430445    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b66997cdf-r8knr"]
Jul 29 22:51:34.491682 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.491645    2580 generic.go:356] "Generic (PLEG): container finished" podID="cece57cc-fa61-43a5-b75a-cfb4e4285af7" containerID="26c00298f8ff6875ad5fb5f7af2af2b1ec46d3cea42bd9a46891e82bfc241389" exitCode=0
Jul 29 22:51:34.491879 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.491737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zb96" event={"ID":"cece57cc-fa61-43a5-b75a-cfb4e4285af7","Type":"ContainerDied","Data":"26c00298f8ff6875ad5fb5f7af2af2b1ec46d3cea42bd9a46891e82bfc241389"}
Jul 29 22:51:34.514149 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514265 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514265 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264210de-4bf3-499e-af03-fd17c0b22984-metrics-client-ca\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-grpc-tls\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgp5s\" (UniqueName: \"kubernetes.io/projected/264210de-4bf3-499e-af03-fd17c0b22984-kube-api-access-dgp5s\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514427 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-tls\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514427 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.514508 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.514432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.615575 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.615544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264210de-4bf3-499e-af03-fd17c0b22984-metrics-client-ca\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.615705 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.615580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-grpc-tls\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.615705 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.615608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgp5s\" (UniqueName: \"kubernetes.io/projected/264210de-4bf3-499e-af03-fd17c0b22984-kube-api-access-dgp5s\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.615906 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.615886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-tls\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.616771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.616164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.616771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.616223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.616771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.616301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.616771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.616366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.616771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.616489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264210de-4bf3-499e-af03-fd17c0b22984-metrics-client-ca\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.618756 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.618708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-grpc-tls\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.618976 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.618953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.619197 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.619173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.619482 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.619461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-tls\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.619673 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.619656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.620341 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.620322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/264210de-4bf3-499e-af03-fd17c0b22984-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.624513 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.624494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgp5s\" (UniqueName: \"kubernetes.io/projected/264210de-4bf3-499e-af03-fd17c0b22984-kube-api-access-dgp5s\") pod \"thanos-querier-b66997cdf-r8knr\" (UID: \"264210de-4bf3-499e-af03-fd17c0b22984\") " pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.723160 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.723079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:34.853378 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:34.853348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b66997cdf-r8knr"]
Jul 29 22:51:34.856176 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:34.856152    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264210de_4bf3_499e_af03_fd17c0b22984.slice/crio-99b6712c4073634e95ffa574ae8eea22d936913a6cc65a2c3639f652ca036d9a WatchSource:0}: Error finding container 99b6712c4073634e95ffa574ae8eea22d936913a6cc65a2c3639f652ca036d9a: Status 404 returned error can't find the container with id 99b6712c4073634e95ffa574ae8eea22d936913a6cc65a2c3639f652ca036d9a
Jul 29 22:51:35.496968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:35.496933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zb96" event={"ID":"cece57cc-fa61-43a5-b75a-cfb4e4285af7","Type":"ContainerStarted","Data":"16ab27f6747e021345e3a70ac594c29acabbf3d6ff30908d9935cacc3851724e"}
Jul 29 22:51:35.497429 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:35.496996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zb96" event={"ID":"cece57cc-fa61-43a5-b75a-cfb4e4285af7","Type":"ContainerStarted","Data":"821f373d297850eb7deb4df1716d55e8fb27b77f1f8f8ad09ad77217b9198be0"}
Jul 29 22:51:35.497992 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:35.497971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"99b6712c4073634e95ffa574ae8eea22d936913a6cc65a2c3639f652ca036d9a"}
Jul 29 22:51:35.520838 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:35.520768    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4zb96" podStartSLOduration=3.546979571 podStartE2EDuration="4.520751625s" totalImagesPullingTime="909.025993ms" totalInitContainerRuntime="64.746061ms" isStatefulPod=true podCreationTimestamp="2026-07-29 22:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:35.519910776 +0000 UTC m=+311.428601131" watchObservedRunningTime="2026-07-29 22:51:35.520751625 +0000 UTC m=+311.429441990"
Jul 29 22:51:36.080080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.080044    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"]
Jul 29 22:51:36.082323 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.082297    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.084591 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.084428    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 22:51:36.084591 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.084463    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8qk6p\""
Jul 29 22:51:36.090533 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.090507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"]
Jul 29 22:51:36.130174 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.130140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwpb\" (UniqueName: \"kubernetes.io/projected/9d2b5b95-5745-41b2-84f2-92d7b9a3f262-kube-api-access-8lwpb\") pod \"monitoring-plugin-68b7fc9574-tg5vj\" (UID: \"9d2b5b95-5745-41b2-84f2-92d7b9a3f262\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.130343 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.130219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d2b5b95-5745-41b2-84f2-92d7b9a3f262-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tg5vj\" (UID: \"9d2b5b95-5745-41b2-84f2-92d7b9a3f262\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.231419 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.231381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwpb\" (UniqueName: \"kubernetes.io/projected/9d2b5b95-5745-41b2-84f2-92d7b9a3f262-kube-api-access-8lwpb\") pod \"monitoring-plugin-68b7fc9574-tg5vj\" (UID: \"9d2b5b95-5745-41b2-84f2-92d7b9a3f262\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.231601 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.231481    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d2b5b95-5745-41b2-84f2-92d7b9a3f262-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tg5vj\" (UID: \"9d2b5b95-5745-41b2-84f2-92d7b9a3f262\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.234330 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.234307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9d2b5b95-5745-41b2-84f2-92d7b9a3f262-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tg5vj\" (UID: \"9d2b5b95-5745-41b2-84f2-92d7b9a3f262\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.239124 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.239100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwpb\" (UniqueName: \"kubernetes.io/projected/9d2b5b95-5745-41b2-84f2-92d7b9a3f262-kube-api-access-8lwpb\") pod \"monitoring-plugin-68b7fc9574-tg5vj\" (UID: \"9d2b5b95-5745-41b2-84f2-92d7b9a3f262\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.394453 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.394363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:36.532232 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:36.532199    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"]
Jul 29 22:51:36.930044 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:36.930011    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2b5b95_5745_41b2_84f2_92d7b9a3f262.slice/crio-42c9852769b068c2f5a3ad87d2dd4814d8227b78430e956d5fa63ae91283de21 WatchSource:0}: Error finding container 42c9852769b068c2f5a3ad87d2dd4814d8227b78430e956d5fa63ae91283de21: Status 404 returned error can't find the container with id 42c9852769b068c2f5a3ad87d2dd4814d8227b78430e956d5fa63ae91283de21
Jul 29 22:51:37.510528 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:37.510477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"4fff26e89536f521ec2ca7fbe88a00ea9ea0c5c12ed262d2dc8b35d17b24e202"}
Jul 29 22:51:37.510710 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:37.510622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"a126664c700f43c04859617d82a31eace591bf2d86eabe831d01b30f09e2a768"}
Jul 29 22:51:37.510782 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:37.510750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"b2c33b28640c9e51302430f474bcdf8338f9bc686f99877f098879f993b6251e"}
Jul 29 22:51:37.511848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:37.511804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj" event={"ID":"9d2b5b95-5745-41b2-84f2-92d7b9a3f262","Type":"ContainerStarted","Data":"42c9852769b068c2f5a3ad87d2dd4814d8227b78430e956d5fa63ae91283de21"}
Jul 29 22:51:37.982407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:37.982362    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6765967669-vw2n9" podUID="68d8774d-2013-49df-9eb5-a0052657dc30" containerName="registry" containerID="cri-o://cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca" gracePeriod=30
Jul 29 22:51:38.278649 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.278628    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bbdbd4446-xztv6"]
Jul 29 22:51:38.280763 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.280737    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:38.281944 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.281926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.284407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.284382    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 22:51:38.284558 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.284535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8pdjv\""
Jul 29 22:51:38.284829 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.284604    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 22:51:38.285065 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.285046    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 22:51:38.285194 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.284625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 22:51:38.285387 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.285272    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 22:51:38.285892 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.285751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 22:51:38.285892 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.285768    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 22:51:38.292500 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.292475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbdbd4446-xztv6"]
Jul 29 22:51:38.348623 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95hnf\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-kube-api-access-95hnf\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348648    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-bound-sa-token\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-registry-certificates\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-installation-pull-secrets\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68d8774d-2013-49df-9eb5-a0052657dc30-ca-trust-extracted\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348834    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-image-registry-private-configuration\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.348880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-trusted-ca\") pod \"68d8774d-2013-49df-9eb5-a0052657dc30\" (UID: \"68d8774d-2013-49df-9eb5-a0052657dc30\") "
Jul 29 22:51:38.349240 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.348967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-console-config\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.349240 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-oauth-serving-cert\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.349240 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-oauth-config\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.349240 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/1abd3781-7d95-48b1-b09a-a134252b2c93-kube-api-access-xvbv8\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.349240 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-service-ca\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.349483 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-serving-cert\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.349483 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:51:38.349634 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.349611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:51:38.351969 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.351929    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:51:38.352066 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.351983    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-kube-api-access-95hnf" (OuterVolumeSpecName: "kube-api-access-95hnf") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "kube-api-access-95hnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:51:38.352123 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.352065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:51:38.352123 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.352088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:51:38.354922 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.354900    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:51:38.358668 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.358645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d8774d-2013-49df-9eb5-a0052657dc30-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "68d8774d-2013-49df-9eb5-a0052657dc30" (UID: "68d8774d-2013-49df-9eb5-a0052657dc30"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:51:38.449976 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.449886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-service-ca\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.449976 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.449943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-serving-cert\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450230 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.449988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-console-config\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450230 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-oauth-serving-cert\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450230 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-oauth-config\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/1abd3781-7d95-48b1-b09a-a134252b2c93-kube-api-access-xvbv8\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450303    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hnf\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-kube-api-access-95hnf\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450314    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-bound-sa-token\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450325    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-registry-certificates\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450334    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-installation-pull-secrets\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450348    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/68d8774d-2013-49df-9eb5-a0052657dc30-ca-trust-extracted\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450361    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/68d8774d-2013-49df-9eb5-a0052657dc30-registry-tls\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450376    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/68d8774d-2013-49df-9eb5-a0052657dc30-image-registry-private-configuration\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450402 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450390    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/68d8774d-2013-49df-9eb5-a0052657dc30-trusted-ca\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:51:38.450845 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-console-config\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450845 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-service-ca\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.450845 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.450778    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-oauth-serving-cert\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.452443 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.452421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-serving-cert\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.452550 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.452426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-oauth-config\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.457499 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.457477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/1abd3781-7d95-48b1-b09a-a134252b2c93-kube-api-access-xvbv8\") pod \"console-7bbdbd4446-xztv6\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") " pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.516298 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.516267    2580 generic.go:356] "Generic (PLEG): container finished" podID="68d8774d-2013-49df-9eb5-a0052657dc30" containerID="cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca" exitCode=0
Jul 29 22:51:38.516463 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.516309    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6765967669-vw2n9"
Jul 29 22:51:38.516463 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.516365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6765967669-vw2n9" event={"ID":"68d8774d-2013-49df-9eb5-a0052657dc30","Type":"ContainerDied","Data":"cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca"}
Jul 29 22:51:38.516463 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.516416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6765967669-vw2n9" event={"ID":"68d8774d-2013-49df-9eb5-a0052657dc30","Type":"ContainerDied","Data":"8b222aa41d2ed43e7ebce0da46620ee30118ff8c66bda79a075f5d0ac84bbde0"}
Jul 29 22:51:38.516463 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.516438    2580 scope.go:117] "RemoveContainer" containerID="cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca"
Jul 29 22:51:38.519858 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.519709    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"be1c45f013fe20cf9998a733d985d4e5f3b542a04983115ad64acde42ef03fdb"}
Jul 29 22:51:38.519858 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.519801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"d42c3d2605fd0fc1a782ba44bb689b7e01e4517b85b69f4eb3c8158d3f451697"}
Jul 29 22:51:38.520047 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.519948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:38.520047 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.520000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" event={"ID":"264210de-4bf3-499e-af03-fd17c0b22984","Type":"ContainerStarted","Data":"610bb070f8a0ed543c0984cfba494a6bd68211da21aa3db14c6a33a7138955fa"}
Jul 29 22:51:38.521221 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.521200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj" event={"ID":"9d2b5b95-5745-41b2-84f2-92d7b9a3f262","Type":"ContainerStarted","Data":"9c8496ef9e1a029a6d2a60efef0b202ecc7589abc584f9176236167609e0c286"}
Jul 29 22:51:38.521385 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.521373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:38.526254 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.526232    2580 scope.go:117] "RemoveContainer" containerID="cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca"
Jul 29 22:51:38.526550 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:51:38.526525    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca\": container with ID starting with cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca not found: ID does not exist" containerID="cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca"
Jul 29 22:51:38.526616 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.526563    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca"} err="failed to get container status \"cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca\": rpc error: code = NotFound desc = could not find container \"cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca\": container with ID starting with cde9324c63285f8a411b926f3c6f96f8e070656d2aadde9f16114a3b05998dca not found: ID does not exist"
Jul 29 22:51:38.526862 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.526848    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj"
Jul 29 22:51:38.542526 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.542480    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr" podStartSLOduration=1.448266628 podStartE2EDuration="4.542465102s" totalImagesPullingTime="3.094198474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:38.54134278 +0000 UTC m=+314.450033138" watchObservedRunningTime="2026-07-29 22:51:38.542465102 +0000 UTC m=+314.451155459"
Jul 29 22:51:38.553011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.552984    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6765967669-vw2n9"]
Jul 29 22:51:38.556906 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.556885    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6765967669-vw2n9"]
Jul 29 22:51:38.568258 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.568228    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d8774d-2013-49df-9eb5-a0052657dc30" path="/var/lib/kubelet/pods/68d8774d-2013-49df-9eb5-a0052657dc30/volumes"
Jul 29 22:51:38.617144 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.617115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:38.741612 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.741520    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tg5vj" podStartSLOduration=1.5002335690000002 podStartE2EDuration="2.7415014s" totalImagesPullingTime="1.241267831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:38.569919226 +0000 UTC m=+314.478609583" watchObservedRunningTime="2026-07-29 22:51:38.7415014 +0000 UTC m=+314.650191757"
Jul 29 22:51:38.742307 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:38.742289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbdbd4446-xztv6"]
Jul 29 22:51:38.744743 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:38.744718    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abd3781_7d95_48b1_b09a_a134252b2c93.slice/crio-2d3efb40bd7bc83cd7f3c3055cba64ddb2d8ea303ad9a2077f8998c6a9f0ce82 WatchSource:0}: Error finding container 2d3efb40bd7bc83cd7f3c3055cba64ddb2d8ea303ad9a2077f8998c6a9f0ce82: Status 404 returned error can't find the container with id 2d3efb40bd7bc83cd7f3c3055cba64ddb2d8ea303ad9a2077f8998c6a9f0ce82
Jul 29 22:51:39.529248 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:39.529207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbdbd4446-xztv6" event={"ID":"1abd3781-7d95-48b1-b09a-a134252b2c93","Type":"ContainerStarted","Data":"2d3efb40bd7bc83cd7f3c3055cba64ddb2d8ea303ad9a2077f8998c6a9f0ce82"}
Jul 29 22:51:40.424661 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.424629    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f44697c48-sjjhw"]
Jul 29 22:51:40.424975 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.424958    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d8774d-2013-49df-9eb5-a0052657dc30" containerName="registry"
Jul 29 22:51:40.425051 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.424977    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d8774d-2013-49df-9eb5-a0052657dc30" containerName="registry"
Jul 29 22:51:40.425108 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.425053    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d8774d-2013-49df-9eb5-a0052657dc30" containerName="registry"
Jul 29 22:51:40.428015 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.427996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.435100 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.435075    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 22:51:40.438157 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.438134    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f44697c48-sjjhw"]
Jul 29 22:51:40.469829 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.469777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-oauth-config\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.469829 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.469833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-config\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.470097 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.469858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-trusted-ca-bundle\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.470097 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.469892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs9vq\" (UniqueName: \"kubernetes.io/projected/5b0fd11e-cdf5-4053-9eff-440036d3fa25-kube-api-access-xs9vq\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.470097 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.469922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-serving-cert\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.470097 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.469992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-oauth-serving-cert\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.470097 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.470020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-service-ca\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.570753 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-serving-cert\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-oauth-serving-cert\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-service-ca\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-oauth-config\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-config\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-trusted-ca-bundle\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.570953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs9vq\" (UniqueName: \"kubernetes.io/projected/5b0fd11e-cdf5-4053-9eff-440036d3fa25-kube-api-access-xs9vq\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.571638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-oauth-serving-cert\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.571996 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.571970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-config\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.572278 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.572236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-trusted-ca-bundle\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.572422 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.572405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-service-ca\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.573750 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.573704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-serving-cert\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.573877 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.573846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-oauth-config\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.582139 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.582087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs9vq\" (UniqueName: \"kubernetes.io/projected/5b0fd11e-cdf5-4053-9eff-440036d3fa25-kube-api-access-xs9vq\") pod \"console-6f44697c48-sjjhw\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") " pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:40.740658 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:40.740585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:41.908090 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:41.908065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f44697c48-sjjhw"]
Jul 29 22:51:41.911032 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:41.911007    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0fd11e_cdf5_4053_9eff_440036d3fa25.slice/crio-93ed9cc356e84c1e945c9062ee1a9e7223c45544bf4c880f19bd8ef141c8c89f WatchSource:0}: Error finding container 93ed9cc356e84c1e945c9062ee1a9e7223c45544bf4c880f19bd8ef141c8c89f: Status 404 returned error can't find the container with id 93ed9cc356e84c1e945c9062ee1a9e7223c45544bf4c880f19bd8ef141c8c89f
Jul 29 22:51:42.543378 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:42.543339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbdbd4446-xztv6" event={"ID":"1abd3781-7d95-48b1-b09a-a134252b2c93","Type":"ContainerStarted","Data":"17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5"}
Jul 29 22:51:42.544775 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:42.544755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44697c48-sjjhw" event={"ID":"5b0fd11e-cdf5-4053-9eff-440036d3fa25","Type":"ContainerStarted","Data":"0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560"}
Jul 29 22:51:42.544867 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:42.544817    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44697c48-sjjhw" event={"ID":"5b0fd11e-cdf5-4053-9eff-440036d3fa25","Type":"ContainerStarted","Data":"93ed9cc356e84c1e945c9062ee1a9e7223c45544bf4c880f19bd8ef141c8c89f"}
Jul 29 22:51:42.559935 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:42.559896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bbdbd4446-xztv6" podStartSLOduration=1.465752844 podStartE2EDuration="4.559882526s" totalImagesPullingTime="3.094129682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:42.558323393 +0000 UTC m=+318.467013747" watchObservedRunningTime="2026-07-29 22:51:42.559882526 +0000 UTC m=+318.468572884"
Jul 29 22:51:42.574213 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:42.574170    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f44697c48-sjjhw" podStartSLOduration=2.574157369 podStartE2EDuration="2.574157369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:42.573302652 +0000 UTC m=+318.481993010" watchObservedRunningTime="2026-07-29 22:51:42.574157369 +0000 UTC m=+318.482847726"
Jul 29 22:51:44.535851 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:44.535818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b66997cdf-r8knr"
Jul 29 22:51:48.617425 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:48.617396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:48.617859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:48.617478    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:48.622054 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:48.622034    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:49.464357 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.464321    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbdbd4446-xztv6"]
Jul 29 22:51:49.489693 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.489664    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b589897bf-fdxk5"]
Jul 29 22:51:49.495168 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.495150    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.502912 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.502883    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b589897bf-fdxk5"]
Jul 29 22:51:49.576078 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.576043    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:51:49.652039 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-service-ca\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.652039 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-console-config\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.652543 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-serving-cert\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.652543 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-oauth-config\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.652543 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652383    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-trusted-ca-bundle\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.652543 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6hd\" (UniqueName: \"kubernetes.io/projected/7f2915ce-ef39-4101-b262-d213de26ff77-kube-api-access-sf6hd\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.652720 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.652554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-oauth-serving-cert\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753728 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-console-config\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753728 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-serving-cert\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753951 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-oauth-config\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753951 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-trusted-ca-bundle\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753951 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6hd\" (UniqueName: \"kubernetes.io/projected/7f2915ce-ef39-4101-b262-d213de26ff77-kube-api-access-sf6hd\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753951 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-oauth-serving-cert\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.753951 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.753868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-service-ca\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.754474 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.754443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-console-config\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.754567 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.754525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-service-ca\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.754618 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.754565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-oauth-serving-cert\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.754618 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.754564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-trusted-ca-bundle\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.756302 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.756283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-oauth-config\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.756387 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.756347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-serving-cert\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.761432 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.761410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6hd\" (UniqueName: \"kubernetes.io/projected/7f2915ce-ef39-4101-b262-d213de26ff77-kube-api-access-sf6hd\") pod \"console-7b589897bf-fdxk5\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") " pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.804496 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.804465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:49.935255 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:49.935224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b589897bf-fdxk5"]
Jul 29 22:51:49.938352 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:51:49.938326    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2915ce_ef39_4101_b262_d213de26ff77.slice/crio-7664d6d93532c5bea243706a945aa75d697ee07af82ed06f250e6660f581c164 WatchSource:0}: Error finding container 7664d6d93532c5bea243706a945aa75d697ee07af82ed06f250e6660f581c164: Status 404 returned error can't find the container with id 7664d6d93532c5bea243706a945aa75d697ee07af82ed06f250e6660f581c164
Jul 29 22:51:50.576929 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:50.576887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b589897bf-fdxk5" event={"ID":"7f2915ce-ef39-4101-b262-d213de26ff77","Type":"ContainerStarted","Data":"c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f"}
Jul 29 22:51:50.577140 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:50.576946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b589897bf-fdxk5" event={"ID":"7f2915ce-ef39-4101-b262-d213de26ff77","Type":"ContainerStarted","Data":"7664d6d93532c5bea243706a945aa75d697ee07af82ed06f250e6660f581c164"}
Jul 29 22:51:50.607783 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:50.607730    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b589897bf-fdxk5" podStartSLOduration=1.6077128969999999 podStartE2EDuration="1.607712897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:51:50.606525053 +0000 UTC m=+326.515215412" watchObservedRunningTime="2026-07-29 22:51:50.607712897 +0000 UTC m=+326.516403254"
Jul 29 22:51:50.741634 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:50.741600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:50.742109 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:50.741722    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:50.746365 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:50.746347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:51.584731 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:51.584705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:51:53.589186 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:53.589155    2580 generic.go:356] "Generic (PLEG): container finished" podID="258f5c2b-06f6-4659-a589-afdbe1c6bf5e" containerID="6a33de7039440c77a61e8a2fcc26c6a181136d3dcc06c5bd7503060b50d1e726" exitCode=0
Jul 29 22:51:53.589663 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:53.589229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dd689" event={"ID":"258f5c2b-06f6-4659-a589-afdbe1c6bf5e","Type":"ContainerDied","Data":"6a33de7039440c77a61e8a2fcc26c6a181136d3dcc06c5bd7503060b50d1e726"}
Jul 29 22:51:53.589735 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:53.589701    2580 scope.go:117] "RemoveContainer" containerID="6a33de7039440c77a61e8a2fcc26c6a181136d3dcc06c5bd7503060b50d1e726"
Jul 29 22:51:54.593860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:54.593824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-dd689" event={"ID":"258f5c2b-06f6-4659-a589-afdbe1c6bf5e","Type":"ContainerStarted","Data":"9661549c62109bf5dabe254518c8a7287d08629eb9239a9c8e7e6ac275ab2043"}
Jul 29 22:51:55.564869 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:55.564843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-tg52h_227b4dbc-e960-4432-b261-f7cf67a14aba/cluster-monitoring-operator/0.log"
Jul 29 22:51:56.563295 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:56.563265    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-tg5vj_9d2b5b95-5745-41b2-84f2-92d7b9a3f262/monitoring-plugin/0.log"
Jul 29 22:51:56.762468 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:56.762439    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zb96_cece57cc-fa61-43a5-b75a-cfb4e4285af7/init-textfile/0.log"
Jul 29 22:51:56.963070 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:56.962974    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zb96_cece57cc-fa61-43a5-b75a-cfb4e4285af7/node-exporter/0.log"
Jul 29 22:51:57.163022 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:57.162990    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zb96_cece57cc-fa61-43a5-b75a-cfb4e4285af7/kube-rbac-proxy/0.log"
Jul 29 22:51:59.805134 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:59.805090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:59.805608 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:59.805162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:51:59.810393 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:51:59.810368    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:52:00.618020 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:00.617993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:52:00.664365 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:00.664332    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f44697c48-sjjhw"]
Jul 29 22:52:01.762807 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:01.762765    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/thanos-query/0.log"
Jul 29 22:52:01.963153 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:01.963123    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy-web/0.log"
Jul 29 22:52:02.162968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:02.162940    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy/0.log"
Jul 29 22:52:02.362885 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:02.362858    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/prom-label-proxy/0.log"
Jul 29 22:52:02.562541 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:02.562518    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy-rules/0.log"
Jul 29 22:52:02.763173 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:02.763143    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy-metrics/0.log"
Jul 29 22:52:03.562978 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:03.562947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f44697c48-sjjhw_5b0fd11e-cdf5-4053-9eff-440036d3fa25/console/0.log"
Jul 29 22:52:03.763623 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:03.763584    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b589897bf-fdxk5_7f2915ce-ef39-4101-b262-d213de26ff77/console/0.log"
Jul 29 22:52:03.962998 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:03.962923    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bbdbd4446-xztv6_1abd3781-7d95-48b1-b09a-a134252b2c93/console/0.log"
Jul 29 22:52:15.595680 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:15.595611    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bbdbd4446-xztv6" podUID="1abd3781-7d95-48b1-b09a-a134252b2c93" containerName="console" containerID="cri-o://17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5" gracePeriod=15
Jul 29 22:52:16.334864 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.334838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:52:16.384637 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.384602    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-oauth-serving-cert\") pod \"1abd3781-7d95-48b1-b09a-a134252b2c93\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") "
Jul 29 22:52:16.384848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.384644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-oauth-config\") pod \"1abd3781-7d95-48b1-b09a-a134252b2c93\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") "
Jul 29 22:52:16.384848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.384685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-serving-cert\") pod \"1abd3781-7d95-48b1-b09a-a134252b2c93\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") "
Jul 29 22:52:16.384848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.384720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-service-ca\") pod \"1abd3781-7d95-48b1-b09a-a134252b2c93\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") "
Jul 29 22:52:16.384848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.384773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-console-config\") pod \"1abd3781-7d95-48b1-b09a-a134252b2c93\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") "
Jul 29 22:52:16.384848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.384823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/1abd3781-7d95-48b1-b09a-a134252b2c93-kube-api-access-xvbv8\") pod \"1abd3781-7d95-48b1-b09a-a134252b2c93\" (UID: \"1abd3781-7d95-48b1-b09a-a134252b2c93\") "
Jul 29 22:52:16.385120 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.385081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1abd3781-7d95-48b1-b09a-a134252b2c93" (UID: "1abd3781-7d95-48b1-b09a-a134252b2c93"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:16.385379 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.385345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-console-config" (OuterVolumeSpecName: "console-config") pod "1abd3781-7d95-48b1-b09a-a134252b2c93" (UID: "1abd3781-7d95-48b1-b09a-a134252b2c93"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:16.385481 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.385389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-service-ca" (OuterVolumeSpecName: "service-ca") pod "1abd3781-7d95-48b1-b09a-a134252b2c93" (UID: "1abd3781-7d95-48b1-b09a-a134252b2c93"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:16.386989 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.386959    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1abd3781-7d95-48b1-b09a-a134252b2c93" (UID: "1abd3781-7d95-48b1-b09a-a134252b2c93"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:16.387086 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.386998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abd3781-7d95-48b1-b09a-a134252b2c93-kube-api-access-xvbv8" (OuterVolumeSpecName: "kube-api-access-xvbv8") pod "1abd3781-7d95-48b1-b09a-a134252b2c93" (UID: "1abd3781-7d95-48b1-b09a-a134252b2c93"). InnerVolumeSpecName "kube-api-access-xvbv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:16.387086 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.387064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1abd3781-7d95-48b1-b09a-a134252b2c93" (UID: "1abd3781-7d95-48b1-b09a-a134252b2c93"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:16.485860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.485759    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-console-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:16.485860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.485810    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/1abd3781-7d95-48b1-b09a-a134252b2c93-kube-api-access-xvbv8\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:16.485860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.485820    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-oauth-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:16.485860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.485830    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-oauth-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:16.485860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.485839    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1abd3781-7d95-48b1-b09a-a134252b2c93-console-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:16.485860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.485847    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1abd3781-7d95-48b1-b09a-a134252b2c93-service-ca\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:16.679233 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.679197    2580 generic.go:356] "Generic (PLEG): container finished" podID="1abd3781-7d95-48b1-b09a-a134252b2c93" containerID="17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5" exitCode=0
Jul 29 22:52:16.679631 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.679248    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbdbd4446-xztv6"
Jul 29 22:52:16.679631 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.679255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbdbd4446-xztv6" event={"ID":"1abd3781-7d95-48b1-b09a-a134252b2c93","Type":"ContainerDied","Data":"17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5"}
Jul 29 22:52:16.679631 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.679308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbdbd4446-xztv6" event={"ID":"1abd3781-7d95-48b1-b09a-a134252b2c93","Type":"ContainerDied","Data":"2d3efb40bd7bc83cd7f3c3055cba64ddb2d8ea303ad9a2077f8998c6a9f0ce82"}
Jul 29 22:52:16.679631 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.679336    2580 scope.go:117] "RemoveContainer" containerID="17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5"
Jul 29 22:52:16.687382 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.687362    2580 scope.go:117] "RemoveContainer" containerID="17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5"
Jul 29 22:52:16.687628 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:52:16.687609    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5\": container with ID starting with 17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5 not found: ID does not exist" containerID="17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5"
Jul 29 22:52:16.687694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.687640    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5"} err="failed to get container status \"17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5\": rpc error: code = NotFound desc = could not find container \"17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5\": container with ID starting with 17ae7ea5db7046ac44883cf1cfc28a7fa33a0a97f4871da99f19a52912083ea5 not found: ID does not exist"
Jul 29 22:52:16.694800 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.694769    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbdbd4446-xztv6"]
Jul 29 22:52:16.698117 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:16.698099    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bbdbd4446-xztv6"]
Jul 29 22:52:18.568502 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:18.568462    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1abd3781-7d95-48b1-b09a-a134252b2c93" path="/var/lib/kubelet/pods/1abd3781-7d95-48b1-b09a-a134252b2c93/volumes"
Jul 29 22:52:23.711143 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:23.711058    2580 generic.go:356] "Generic (PLEG): container finished" podID="93aa25fa-17e8-49c4-aad3-2fcc8d472c73" containerID="9a38a766439ca2eb3f8542c5e376d4a4dcbf15285430be31f9d63befc54e6b79" exitCode=0
Jul 29 22:52:23.711614 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:23.711134    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm" event={"ID":"93aa25fa-17e8-49c4-aad3-2fcc8d472c73","Type":"ContainerDied","Data":"9a38a766439ca2eb3f8542c5e376d4a4dcbf15285430be31f9d63befc54e6b79"}
Jul 29 22:52:23.711614 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:23.711530    2580 scope.go:117] "RemoveContainer" containerID="9a38a766439ca2eb3f8542c5e376d4a4dcbf15285430be31f9d63befc54e6b79"
Jul 29 22:52:24.716993 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:24.716911    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-htbcm" event={"ID":"93aa25fa-17e8-49c4-aad3-2fcc8d472c73","Type":"ContainerStarted","Data":"0207b948e76d19b0ad65ca63a62efb4becec9130dfbc04eaf6b9584bd2ba7afe"}
Jul 29 22:52:25.684384 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:25.684335    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f44697c48-sjjhw" podUID="5b0fd11e-cdf5-4053-9eff-440036d3fa25" containerName="console" containerID="cri-o://0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560" gracePeriod=15
Jul 29 22:52:25.953662 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:25.953640    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:52:26.068459 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-oauth-serving-cert\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.068655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs9vq\" (UniqueName: \"kubernetes.io/projected/5b0fd11e-cdf5-4053-9eff-440036d3fa25-kube-api-access-xs9vq\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.068655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-config\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.068655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068511    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-trusted-ca-bundle\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.068655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-service-ca\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.068655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-serving-cert\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.068655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.068601    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-oauth-config\") pod \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\" (UID: \"5b0fd11e-cdf5-4053-9eff-440036d3fa25\") "
Jul 29 22:52:26.069038 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.069010    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-config" (OuterVolumeSpecName: "console-config") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:26.069090 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.069041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:26.069090 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.069015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:26.069207 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.069189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:52:26.070769 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.070742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:26.071246 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.071225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:52:26.071305 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.071234    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0fd11e-cdf5-4053-9eff-440036d3fa25-kube-api-access-xs9vq" (OuterVolumeSpecName: "kube-api-access-xs9vq") pod "5b0fd11e-cdf5-4053-9eff-440036d3fa25" (UID: "5b0fd11e-cdf5-4053-9eff-440036d3fa25"). InnerVolumeSpecName "kube-api-access-xs9vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:52:26.169910 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169879    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-service-ca\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.169910 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169903    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.169910 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169913    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-oauth-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.170114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169923    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-oauth-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.170114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169932    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs9vq\" (UniqueName: \"kubernetes.io/projected/5b0fd11e-cdf5-4053-9eff-440036d3fa25-kube-api-access-xs9vq\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.170114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169940    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-console-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.170114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.169948    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b0fd11e-cdf5-4053-9eff-440036d3fa25-trusted-ca-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:52:26.725219 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.725185    2580 generic.go:356] "Generic (PLEG): container finished" podID="5b0fd11e-cdf5-4053-9eff-440036d3fa25" containerID="0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560" exitCode=0
Jul 29 22:52:26.725379 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.725227    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44697c48-sjjhw"
Jul 29 22:52:26.725379 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.725250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44697c48-sjjhw" event={"ID":"5b0fd11e-cdf5-4053-9eff-440036d3fa25","Type":"ContainerDied","Data":"0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560"}
Jul 29 22:52:26.725379 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.725295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44697c48-sjjhw" event={"ID":"5b0fd11e-cdf5-4053-9eff-440036d3fa25","Type":"ContainerDied","Data":"93ed9cc356e84c1e945c9062ee1a9e7223c45544bf4c880f19bd8ef141c8c89f"}
Jul 29 22:52:26.725379 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.725316    2580 scope.go:117] "RemoveContainer" containerID="0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560"
Jul 29 22:52:26.733214 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.733195    2580 scope.go:117] "RemoveContainer" containerID="0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560"
Jul 29 22:52:26.733477 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:52:26.733456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560\": container with ID starting with 0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560 not found: ID does not exist" containerID="0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560"
Jul 29 22:52:26.733562 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.733482    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560"} err="failed to get container status \"0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560\": rpc error: code = NotFound desc = could not find container \"0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560\": container with ID starting with 0cd0ed21a813e5777235e7adc9f935f4a8e3240ab7d38150edb3314d7446d560 not found: ID does not exist"
Jul 29 22:52:26.739699 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.739669    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f44697c48-sjjhw"]
Jul 29 22:52:26.744636 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:26.744617    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f44697c48-sjjhw"]
Jul 29 22:52:28.568307 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:28.568280    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0fd11e-cdf5-4053-9eff-440036d3fa25" path="/var/lib/kubelet/pods/5b0fd11e-cdf5-4053-9eff-440036d3fa25/volumes"
Jul 29 22:52:38.371183 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:38.371137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:52:38.373539 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:38.373514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af595ab0-8515-4923-af82-6cdcd20b5c57-metrics-certs\") pod \"network-metrics-daemon-vct9g\" (UID: \"af595ab0-8515-4923-af82-6cdcd20b5c57\") " pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:52:38.566060 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:38.566030    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wkbjs\""
Jul 29 22:52:38.573908 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:38.573887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vct9g"
Jul 29 22:52:38.699747 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:38.699715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vct9g"]
Jul 29 22:52:38.702043 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:52:38.702012    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf595ab0_8515_4923_af82_6cdcd20b5c57.slice/crio-a4433d0b2c4ddfef25b19c2dd6d779a97af34a4ce6e748d475930d6d929fa5c6 WatchSource:0}: Error finding container a4433d0b2c4ddfef25b19c2dd6d779a97af34a4ce6e748d475930d6d929fa5c6: Status 404 returned error can't find the container with id a4433d0b2c4ddfef25b19c2dd6d779a97af34a4ce6e748d475930d6d929fa5c6
Jul 29 22:52:38.772325 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:38.772295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vct9g" event={"ID":"af595ab0-8515-4923-af82-6cdcd20b5c57","Type":"ContainerStarted","Data":"a4433d0b2c4ddfef25b19c2dd6d779a97af34a4ce6e748d475930d6d929fa5c6"}
Jul 29 22:52:39.777894 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:39.777858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vct9g" event={"ID":"af595ab0-8515-4923-af82-6cdcd20b5c57","Type":"ContainerStarted","Data":"6446e8e53a8996a69e361e2712bce2f7d1d343bed0bc9a9c9f9515b0eb497e59"}
Jul 29 22:52:39.778263 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:39.777922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vct9g" event={"ID":"af595ab0-8515-4923-af82-6cdcd20b5c57","Type":"ContainerStarted","Data":"cbfcf5e3dc23293b8b7056d16883f18699d3fc3f7cdddc67a18db0a8f4a11edf"}
Jul 29 22:52:39.793937 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:39.793886    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vct9g" podStartSLOduration=374.941707552 podStartE2EDuration="6m15.793868102s" totalImagesPullingTime="852.16055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:52:39.792011708 +0000 UTC m=+375.700702066" watchObservedRunningTime="2026-07-29 22:52:39.793868102 +0000 UTC m=+375.702558460"
Jul 29 22:52:56.227284 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227251    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5985c6b55-g45lj"]
Jul 29 22:52:56.227697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227526    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1abd3781-7d95-48b1-b09a-a134252b2c93" containerName="console"
Jul 29 22:52:56.227697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227536    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1abd3781-7d95-48b1-b09a-a134252b2c93" containerName="console"
Jul 29 22:52:56.227697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227544    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0fd11e-cdf5-4053-9eff-440036d3fa25" containerName="console"
Jul 29 22:52:56.227697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0fd11e-cdf5-4053-9eff-440036d3fa25" containerName="console"
Jul 29 22:52:56.227697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227591    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1abd3781-7d95-48b1-b09a-a134252b2c93" containerName="console"
Jul 29 22:52:56.227697 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.227597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0fd11e-cdf5-4053-9eff-440036d3fa25" containerName="console"
Jul 29 22:52:56.229641 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.229620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.241185 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.241163    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5985c6b55-g45lj"]
Jul 29 22:52:56.416934 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.416899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-trusted-ca-bundle\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.417112 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.416940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-oauth-config\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.417112 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.416997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5cv\" (UniqueName: \"kubernetes.io/projected/b90c6aa2-a7a7-4785-a399-23028bc39cad-kube-api-access-xx5cv\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.417112 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.417030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-config\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.417112 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.417060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-service-ca\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.417294 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.417117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-serving-cert\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.417294 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.417163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-oauth-serving-cert\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.517724 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.517641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-oauth-serving-cert\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.517724 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.517684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-trusted-ca-bundle\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.517724 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.517711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-oauth-config\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.517986 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.517741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5cv\" (UniqueName: \"kubernetes.io/projected/b90c6aa2-a7a7-4785-a399-23028bc39cad-kube-api-access-xx5cv\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.517986 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.517773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-config\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.517986 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.517922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-service-ca\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.518135 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.518017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-serving-cert\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.518495 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.518467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-oauth-serving-cert\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.518495 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.518485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-config\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.518624 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.518592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-trusted-ca-bundle\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.518662 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.518622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-service-ca\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.520253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.520228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-oauth-config\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.520357 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.520317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-serving-cert\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.525019 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.524999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5cv\" (UniqueName: \"kubernetes.io/projected/b90c6aa2-a7a7-4785-a399-23028bc39cad-kube-api-access-xx5cv\") pod \"console-5985c6b55-g45lj\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") " pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.538994 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.538973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:52:56.666646 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.666621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5985c6b55-g45lj"]
Jul 29 22:52:56.669101 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:52:56.669072    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90c6aa2_a7a7_4785_a399_23028bc39cad.slice/crio-79f039cac8240bb2ae0c4120cc92f5b7534e5f3b7bbacea614b0525c64377e0f WatchSource:0}: Error finding container 79f039cac8240bb2ae0c4120cc92f5b7534e5f3b7bbacea614b0525c64377e0f: Status 404 returned error can't find the container with id 79f039cac8240bb2ae0c4120cc92f5b7534e5f3b7bbacea614b0525c64377e0f
Jul 29 22:52:56.841333 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.841298    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5985c6b55-g45lj" event={"ID":"b90c6aa2-a7a7-4785-a399-23028bc39cad","Type":"ContainerStarted","Data":"e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01"}
Jul 29 22:52:56.841501 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.841349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5985c6b55-g45lj" event={"ID":"b90c6aa2-a7a7-4785-a399-23028bc39cad","Type":"ContainerStarted","Data":"79f039cac8240bb2ae0c4120cc92f5b7534e5f3b7bbacea614b0525c64377e0f"}
Jul 29 22:52:56.858326 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:52:56.858269    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5985c6b55-g45lj" podStartSLOduration=0.858253808 podStartE2EDuration="858.253808ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:52:56.855958886 +0000 UTC m=+392.764649244" watchObservedRunningTime="2026-07-29 22:52:56.858253808 +0000 UTC m=+392.766944166"
Jul 29 22:53:06.540052 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:06.540005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:53:06.540446 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:06.540090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:53:06.544863 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:06.544839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:53:06.882133 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:06.882108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:53:06.930646 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:06.930609    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b589897bf-fdxk5"]
Jul 29 22:53:31.954494 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:31.954440    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b589897bf-fdxk5" podUID="7f2915ce-ef39-4101-b262-d213de26ff77" containerName="console" containerID="cri-o://c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f" gracePeriod=15
Jul 29 22:53:32.194580 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.194557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:53:32.280274 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280192    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-trusted-ca-bundle\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280274 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280228    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-oauth-serving-cert\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280274 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-oauth-config\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280584 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-serving-cert\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280584 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280312    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-service-ca\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280584 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-console-config\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280584 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf6hd\" (UniqueName: \"kubernetes.io/projected/7f2915ce-ef39-4101-b262-d213de26ff77-kube-api-access-sf6hd\") pod \"7f2915ce-ef39-4101-b262-d213de26ff77\" (UID: \"7f2915ce-ef39-4101-b262-d213de26ff77\") "
Jul 29 22:53:32.280781 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:53:32.280869 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280826    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:53:32.280921 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280878    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:53:32.280973 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.280919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-console-config" (OuterVolumeSpecName: "console-config") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:53:32.282464 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.282437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:53:32.282464 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.282454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:53:32.282606 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.282494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2915ce-ef39-4101-b262-d213de26ff77-kube-api-access-sf6hd" (OuterVolumeSpecName: "kube-api-access-sf6hd") pod "7f2915ce-ef39-4101-b262-d213de26ff77" (UID: "7f2915ce-ef39-4101-b262-d213de26ff77"). InnerVolumeSpecName "kube-api-access-sf6hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:53:32.381443 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381410    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-trusted-ca-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.381443 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381435    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-oauth-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.381443 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381444    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-oauth-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.381443 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381452    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f2915ce-ef39-4101-b262-d213de26ff77-console-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.381696 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381461    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-service-ca\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.381696 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381470    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f2915ce-ef39-4101-b262-d213de26ff77-console-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.381696 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.381478    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6hd\" (UniqueName: \"kubernetes.io/projected/7f2915ce-ef39-4101-b262-d213de26ff77-kube-api-access-sf6hd\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:53:32.968353 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.968267    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f2915ce-ef39-4101-b262-d213de26ff77" containerID="c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f" exitCode=0
Jul 29 22:53:32.968353 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.968316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b589897bf-fdxk5"
Jul 29 22:53:32.968778 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.968349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b589897bf-fdxk5" event={"ID":"7f2915ce-ef39-4101-b262-d213de26ff77","Type":"ContainerDied","Data":"c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f"}
Jul 29 22:53:32.968778 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.968403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b589897bf-fdxk5" event={"ID":"7f2915ce-ef39-4101-b262-d213de26ff77","Type":"ContainerDied","Data":"7664d6d93532c5bea243706a945aa75d697ee07af82ed06f250e6660f581c164"}
Jul 29 22:53:32.968778 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.968423    2580 scope.go:117] "RemoveContainer" containerID="c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f"
Jul 29 22:53:32.976764 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.976748    2580 scope.go:117] "RemoveContainer" containerID="c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f"
Jul 29 22:53:32.977049 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:53:32.977030    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f\": container with ID starting with c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f not found: ID does not exist" containerID="c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f"
Jul 29 22:53:32.977127 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.977059    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f"} err="failed to get container status \"c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f\": rpc error: code = NotFound desc = could not find container \"c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f\": container with ID starting with c88ae9a1861b3910fbecc484142ce56b2050b7f40431939981390b48bd3e737f not found: ID does not exist"
Jul 29 22:53:32.982526 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.982507    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b589897bf-fdxk5"]
Jul 29 22:53:32.985176 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:32.985158    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b589897bf-fdxk5"]
Jul 29 22:53:34.571914 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:53:34.571878    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2915ce-ef39-4101-b262-d213de26ff77" path="/var/lib/kubelet/pods/7f2915ce-ef39-4101-b262-d213de26ff77/volumes"
Jul 29 22:54:18.328087 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.328055    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"]
Jul 29 22:54:18.328596 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.328340    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2915ce-ef39-4101-b262-d213de26ff77" containerName="console"
Jul 29 22:54:18.328596 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.328350    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2915ce-ef39-4101-b262-d213de26ff77" containerName="console"
Jul 29 22:54:18.328596 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.328396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2915ce-ef39-4101-b262-d213de26ff77" containerName="console"
Jul 29 22:54:18.331281 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.331265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.333388 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.333366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 22:54:18.333968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.333944    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 22:54:18.333968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.333948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n86fd\""
Jul 29 22:54:18.339566 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.339545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"]
Jul 29 22:54:18.440992 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.440934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.440992 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.440992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.441213 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.441080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf654\" (UniqueName: \"kubernetes.io/projected/cbf60375-16a1-4e77-8368-b9bf390a543c-kube-api-access-wf654\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.541705 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.541662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.541705 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.541713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.541973 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.541754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf654\" (UniqueName: \"kubernetes.io/projected/cbf60375-16a1-4e77-8368-b9bf390a543c-kube-api-access-wf654\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.542145 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.542119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.542212 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.542125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.549531 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.549504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf654\" (UniqueName: \"kubernetes.io/projected/cbf60375-16a1-4e77-8368-b9bf390a543c-kube-api-access-wf654\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.641337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.641239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:18.769426 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:18.769403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"]
Jul 29 22:54:18.772103 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:54:18.772072    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf60375_16a1_4e77_8368_b9bf390a543c.slice/crio-b35e038d4c85477b03b7c19e428db70e39fed93329d2c422714750b1743d9793 WatchSource:0}: Error finding container b35e038d4c85477b03b7c19e428db70e39fed93329d2c422714750b1743d9793: Status 404 returned error can't find the container with id b35e038d4c85477b03b7c19e428db70e39fed93329d2c422714750b1743d9793
Jul 29 22:54:19.137847 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:19.137813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz" event={"ID":"cbf60375-16a1-4e77-8368-b9bf390a543c","Type":"ContainerStarted","Data":"b35e038d4c85477b03b7c19e428db70e39fed93329d2c422714750b1743d9793"}
Jul 29 22:54:26.165559 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:26.165526    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerID="0f1633fc408341f76e8107e9c1a48bbfd0de6a27a43c14080703b06eaaf8fc0f" exitCode=0
Jul 29 22:54:26.165961 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:26.165609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz" event={"ID":"cbf60375-16a1-4e77-8368-b9bf390a543c","Type":"ContainerDied","Data":"0f1633fc408341f76e8107e9c1a48bbfd0de6a27a43c14080703b06eaaf8fc0f"}
Jul 29 22:54:28.174970 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:28.174940    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerID="f1523afe3510d3319b6961cd683001db33318b40d09d11ebea8824c0ed0b9b91" exitCode=0
Jul 29 22:54:28.175283 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:28.175023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz" event={"ID":"cbf60375-16a1-4e77-8368-b9bf390a543c","Type":"ContainerDied","Data":"f1523afe3510d3319b6961cd683001db33318b40d09d11ebea8824c0ed0b9b91"}
Jul 29 22:54:35.206562 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:35.206525    2580 generic.go:356] "Generic (PLEG): container finished" podID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerID="33c81116781f68c38b634db3914d50d6a9ddfebc8b63d573b044afbdea2f3131" exitCode=0
Jul 29 22:54:35.206968 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:35.206623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz" event={"ID":"cbf60375-16a1-4e77-8368-b9bf390a543c","Type":"ContainerDied","Data":"33c81116781f68c38b634db3914d50d6a9ddfebc8b63d573b044afbdea2f3131"}
Jul 29 22:54:36.341414 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.341391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:36.375049 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.375022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-util\") pod \"cbf60375-16a1-4e77-8368-b9bf390a543c\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") "
Jul 29 22:54:36.375222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.375064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-bundle\") pod \"cbf60375-16a1-4e77-8368-b9bf390a543c\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") "
Jul 29 22:54:36.375222 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.375083    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf654\" (UniqueName: \"kubernetes.io/projected/cbf60375-16a1-4e77-8368-b9bf390a543c-kube-api-access-wf654\") pod \"cbf60375-16a1-4e77-8368-b9bf390a543c\" (UID: \"cbf60375-16a1-4e77-8368-b9bf390a543c\") "
Jul 29 22:54:36.375747 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.375724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-bundle" (OuterVolumeSpecName: "bundle") pod "cbf60375-16a1-4e77-8368-b9bf390a543c" (UID: "cbf60375-16a1-4e77-8368-b9bf390a543c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:54:36.377518 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.377484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf60375-16a1-4e77-8368-b9bf390a543c-kube-api-access-wf654" (OuterVolumeSpecName: "kube-api-access-wf654") pod "cbf60375-16a1-4e77-8368-b9bf390a543c" (UID: "cbf60375-16a1-4e77-8368-b9bf390a543c"). InnerVolumeSpecName "kube-api-access-wf654". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:54:36.379159 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.379132    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-util" (OuterVolumeSpecName: "util") pod "cbf60375-16a1-4e77-8368-b9bf390a543c" (UID: "cbf60375-16a1-4e77-8368-b9bf390a543c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:54:36.475883 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.475765    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:54:36.475883 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.475830    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbf60375-16a1-4e77-8368-b9bf390a543c-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:54:36.475883 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:36.475843    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf654\" (UniqueName: \"kubernetes.io/projected/cbf60375-16a1-4e77-8368-b9bf390a543c-kube-api-access-wf654\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:54:37.215884 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:37.215841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz" event={"ID":"cbf60375-16a1-4e77-8368-b9bf390a543c","Type":"ContainerDied","Data":"b35e038d4c85477b03b7c19e428db70e39fed93329d2c422714750b1743d9793"}
Jul 29 22:54:37.216057 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:37.215899    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b35e038d4c85477b03b7c19e428db70e39fed93329d2c422714750b1743d9793"
Jul 29 22:54:37.216057 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:37.215931    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5xjqkz"
Jul 29 22:54:40.750468 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750436    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"]
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750711    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="util"
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750723    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="util"
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750731    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="pull"
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750736    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="pull"
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750746    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="extract"
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750752    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="extract"
Jul 29 22:54:40.750859 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.750811    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf60375-16a1-4e77-8368-b9bf390a543c" containerName="extract"
Jul 29 22:54:40.790490 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.790455    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"]
Jul 29 22:54:40.790630 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.790572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:40.793069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.793016    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 29 22:54:40.793069 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.793060    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-79hnw\""
Jul 29 22:54:40.793258 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.793085    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 29 22:54:40.793258 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.793135    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 29 22:54:40.809019 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.808988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/01a23860-1943-41c9-90a3-452543aabb70-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk\" (UID: \"01a23860-1943-41c9-90a3-452543aabb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:40.809019 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.809023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nnh\" (UniqueName: \"kubernetes.io/projected/01a23860-1943-41c9-90a3-452543aabb70-kube-api-access-86nnh\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk\" (UID: \"01a23860-1943-41c9-90a3-452543aabb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:40.909803 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.909751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/01a23860-1943-41c9-90a3-452543aabb70-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk\" (UID: \"01a23860-1943-41c9-90a3-452543aabb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:40.909986 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.909812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nnh\" (UniqueName: \"kubernetes.io/projected/01a23860-1943-41c9-90a3-452543aabb70-kube-api-access-86nnh\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk\" (UID: \"01a23860-1943-41c9-90a3-452543aabb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:40.912000 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.911978    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/01a23860-1943-41c9-90a3-452543aabb70-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk\" (UID: \"01a23860-1943-41c9-90a3-452543aabb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:40.924959 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:40.924929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nnh\" (UniqueName: \"kubernetes.io/projected/01a23860-1943-41c9-90a3-452543aabb70-kube-api-access-86nnh\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk\" (UID: \"01a23860-1943-41c9-90a3-452543aabb70\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:41.100247 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:41.100208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:41.257098 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:41.257069    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"]
Jul 29 22:54:41.260215 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:54:41.260182    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a23860_1943_41c9_90a3_452543aabb70.slice/crio-f89353ede8772ac811287b1021308992c418fd68f1a58a70264c878bd43bd6a8 WatchSource:0}: Error finding container f89353ede8772ac811287b1021308992c418fd68f1a58a70264c878bd43bd6a8: Status 404 returned error can't find the container with id f89353ede8772ac811287b1021308992c418fd68f1a58a70264c878bd43bd6a8
Jul 29 22:54:42.237230 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:42.237194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk" event={"ID":"01a23860-1943-41c9-90a3-452543aabb70","Type":"ContainerStarted","Data":"f89353ede8772ac811287b1021308992c418fd68f1a58a70264c878bd43bd6a8"}
Jul 29 22:54:45.168833 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.168799    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-rcwgh"]
Jul 29 22:54:45.188391 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.188358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-rcwgh"]
Jul 29 22:54:45.188549 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.188466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.190532 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.190512    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 29 22:54:45.190532 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.190512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 29 22:54:45.190706 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.190567    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-74tcb\""
Jul 29 22:54:45.248653 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.248620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6d5\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-kube-api-access-7n6d5\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.248850 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.248666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2cff07be-c21a-414a-b852-26a58cfb581c-cabundle0\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.248850 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.248770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-certificates\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.251057 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.251027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk" event={"ID":"01a23860-1943-41c9-90a3-452543aabb70","Type":"ContainerStarted","Data":"4d0b6f0dcfebce081ae33a56dfcfb02c8fc39d1d8e87de89c3b2fe552b213b3a"}
Jul 29 22:54:45.251185 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.251121    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:54:45.276063 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.276012    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk" podStartSLOduration=2.163510758 podStartE2EDuration="5.275992004s" totalImagesPullingTime="3.112481246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:54:45.274749476 +0000 UTC m=+501.183439835" watchObservedRunningTime="2026-07-29 22:54:45.275992004 +0000 UTC m=+501.184682363"
Jul 29 22:54:45.349976 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.349945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-certificates\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.350150 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.350032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6d5\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-kube-api-access-7n6d5\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.350150 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.350069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2cff07be-c21a-414a-b852-26a58cfb581c-cabundle0\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.350150 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:54:45.350112    2580 secret.go:281] references non-existent secret key: ca.crt
Jul 29 22:54:45.350150 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:54:45.350137    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 29 22:54:45.350150 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:54:45.350149    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-rcwgh: references non-existent secret key: ca.crt
Jul 29 22:54:45.350343 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:54:45.350246    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-certificates podName:2cff07be-c21a-414a-b852-26a58cfb581c nodeName:}" failed. No retries permitted until 2026-07-29 22:54:45.85022801 +0000 UTC m=+501.758918363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-certificates") pod "keda-operator-77f648d694-rcwgh" (UID: "2cff07be-c21a-414a-b852-26a58cfb581c") : references non-existent secret key: ca.crt
Jul 29 22:54:45.350712 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.350695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2cff07be-c21a-414a-b852-26a58cfb581c-cabundle0\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.359372 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.359351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6d5\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-kube-api-access-7n6d5\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.853834 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.853782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-certificates\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.856143 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.856122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2cff07be-c21a-414a-b852-26a58cfb581c-certificates\") pod \"keda-operator-77f648d694-rcwgh\" (UID: \"2cff07be-c21a-414a-b852-26a58cfb581c\") " pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:45.925281 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.925247    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-lqc9h"]
Jul 29 22:54:45.946014 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.945979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-lqc9h"]
Jul 29 22:54:45.946198 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.946099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:45.948132 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:45.948112    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 29 22:54:46.055681 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.055650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdwm\" (UniqueName: \"kubernetes.io/projected/8c3ba697-0ab4-48b3-95d2-153054c8af02-kube-api-access-6xdwm\") pod \"keda-admission-56f9798bb4-lqc9h\" (UID: \"8c3ba697-0ab4-48b3-95d2-153054c8af02\") " pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.055880 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.055704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c3ba697-0ab4-48b3-95d2-153054c8af02-certificates\") pod \"keda-admission-56f9798bb4-lqc9h\" (UID: \"8c3ba697-0ab4-48b3-95d2-153054c8af02\") " pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.098827 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.098770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:46.156756 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.156728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdwm\" (UniqueName: \"kubernetes.io/projected/8c3ba697-0ab4-48b3-95d2-153054c8af02-kube-api-access-6xdwm\") pod \"keda-admission-56f9798bb4-lqc9h\" (UID: \"8c3ba697-0ab4-48b3-95d2-153054c8af02\") " pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.156935 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.156777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c3ba697-0ab4-48b3-95d2-153054c8af02-certificates\") pod \"keda-admission-56f9798bb4-lqc9h\" (UID: \"8c3ba697-0ab4-48b3-95d2-153054c8af02\") " pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.159350 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.159324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c3ba697-0ab4-48b3-95d2-153054c8af02-certificates\") pod \"keda-admission-56f9798bb4-lqc9h\" (UID: \"8c3ba697-0ab4-48b3-95d2-153054c8af02\") " pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.165617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.165583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdwm\" (UniqueName: \"kubernetes.io/projected/8c3ba697-0ab4-48b3-95d2-153054c8af02-kube-api-access-6xdwm\") pod \"keda-admission-56f9798bb4-lqc9h\" (UID: \"8c3ba697-0ab4-48b3-95d2-153054c8af02\") " pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.250530 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.250505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-rcwgh"]
Jul 29 22:54:46.253030 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:54:46.253002    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cff07be_c21a_414a_b852_26a58cfb581c.slice/crio-e8e3ee4429696c766c63513b497bf6fc03c6941bd07bf84eae59104926042fd6 WatchSource:0}: Error finding container e8e3ee4429696c766c63513b497bf6fc03c6941bd07bf84eae59104926042fd6: Status 404 returned error can't find the container with id e8e3ee4429696c766c63513b497bf6fc03c6941bd07bf84eae59104926042fd6
Jul 29 22:54:46.255798 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.255773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:46.416306 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:46.416230    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-lqc9h"]
Jul 29 22:54:46.418751 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:54:46.418725    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ba697_0ab4_48b3_95d2_153054c8af02.slice/crio-68e2281b81a900541bd6e1e73e39e12fe00d8f49c4f3ce2f908a150ba5b396e9 WatchSource:0}: Error finding container 68e2281b81a900541bd6e1e73e39e12fe00d8f49c4f3ce2f908a150ba5b396e9: Status 404 returned error can't find the container with id 68e2281b81a900541bd6e1e73e39e12fe00d8f49c4f3ce2f908a150ba5b396e9
Jul 29 22:54:47.261799 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:47.261759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h" event={"ID":"8c3ba697-0ab4-48b3-95d2-153054c8af02","Type":"ContainerStarted","Data":"68e2281b81a900541bd6e1e73e39e12fe00d8f49c4f3ce2f908a150ba5b396e9"}
Jul 29 22:54:47.263312 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:47.263263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-rcwgh" event={"ID":"2cff07be-c21a-414a-b852-26a58cfb581c","Type":"ContainerStarted","Data":"e8e3ee4429696c766c63513b497bf6fc03c6941bd07bf84eae59104926042fd6"}
Jul 29 22:54:50.280333 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:50.280300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h" event={"ID":"8c3ba697-0ab4-48b3-95d2-153054c8af02","Type":"ContainerStarted","Data":"6a821632d2859abb2c97418e8ab2b3b443a2441ea2cfd429b945517f67a47422"}
Jul 29 22:54:50.280812 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:50.280378    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:54:50.281864 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:50.281834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-rcwgh" event={"ID":"2cff07be-c21a-414a-b852-26a58cfb581c","Type":"ContainerStarted","Data":"349eb1ba16b975004ff095b30f6ed0ee02ebb82c3c014cad3fef489d9e6bd762"}
Jul 29 22:54:50.282005 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:50.281993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:54:50.295799 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:50.295741    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h" podStartSLOduration=2.068835134 podStartE2EDuration="5.295726873s" totalImagesPullingTime="3.226891739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:54:50.295217798 +0000 UTC m=+506.203908178" watchObservedRunningTime="2026-07-29 22:54:50.295726873 +0000 UTC m=+506.204417231"
Jul 29 22:54:50.310622 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:54:50.310576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-rcwgh" podStartSLOduration=1.882582883 podStartE2EDuration="5.310561854s" totalImagesPullingTime="3.427978971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:54:50.309720517 +0000 UTC m=+506.218410872" watchObservedRunningTime="2026-07-29 22:54:50.310561854 +0000 UTC m=+506.219252213"
Jul 29 22:55:06.257643 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:55:06.257608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jvbzk"
Jul 29 22:55:11.288719 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:55:11.288686    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-rcwgh"
Jul 29 22:55:11.289190 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:55:11.288764    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-lqc9h"
Jul 29 22:56:03.860764 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.860732    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-844cf74686-qq7rx"]
Jul 29 22:56:03.863242 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.863218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:03.865200 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.865169    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 29 22:56:03.865200 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.865176    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 22:56:03.865375 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.865176    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 22:56:03.865375 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.865225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-vxh4w\""
Jul 29 22:56:03.874530 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.874509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-qq7rx"]
Jul 29 22:56:03.909021 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.908994    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qvr5s"]
Jul 29 22:56:03.911464 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.911449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:03.913294 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.913277    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 22:56:03.913377 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.913360    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5sh48\""
Jul 29 22:56:03.918926 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.918904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qvr5s"]
Jul 29 22:56:03.985997 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.985960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbjt\" (UniqueName: \"kubernetes.io/projected/47088f3b-a2d1-4d78-bb79-7c969fdaeb65-kube-api-access-wdbjt\") pod \"seaweedfs-64568bcd49-qvr5s\" (UID: \"47088f3b-a2d1-4d78-bb79-7c969fdaeb65\") " pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:03.985997 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.986004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/47088f3b-a2d1-4d78-bb79-7c969fdaeb65-data\") pod \"seaweedfs-64568bcd49-qvr5s\" (UID: \"47088f3b-a2d1-4d78-bb79-7c969fdaeb65\") " pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:03.986229 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.986039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-cert\") pod \"kserve-controller-manager-844cf74686-qq7rx\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") " pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:03.986229 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:03.986058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hd4\" (UniqueName: \"kubernetes.io/projected/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-kube-api-access-b2hd4\") pod \"kserve-controller-manager-844cf74686-qq7rx\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") " pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:04.087494 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.087454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-cert\") pod \"kserve-controller-manager-844cf74686-qq7rx\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") " pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:04.087494 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.087492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hd4\" (UniqueName: \"kubernetes.io/projected/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-kube-api-access-b2hd4\") pod \"kserve-controller-manager-844cf74686-qq7rx\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") " pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:04.087694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.087554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbjt\" (UniqueName: \"kubernetes.io/projected/47088f3b-a2d1-4d78-bb79-7c969fdaeb65-kube-api-access-wdbjt\") pod \"seaweedfs-64568bcd49-qvr5s\" (UID: \"47088f3b-a2d1-4d78-bb79-7c969fdaeb65\") " pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:04.087694 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.087572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/47088f3b-a2d1-4d78-bb79-7c969fdaeb65-data\") pod \"seaweedfs-64568bcd49-qvr5s\" (UID: \"47088f3b-a2d1-4d78-bb79-7c969fdaeb65\") " pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:04.087962 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.087931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/47088f3b-a2d1-4d78-bb79-7c969fdaeb65-data\") pod \"seaweedfs-64568bcd49-qvr5s\" (UID: \"47088f3b-a2d1-4d78-bb79-7c969fdaeb65\") " pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:04.090064 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.090029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-cert\") pod \"kserve-controller-manager-844cf74686-qq7rx\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") " pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:04.096643 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.096621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbjt\" (UniqueName: \"kubernetes.io/projected/47088f3b-a2d1-4d78-bb79-7c969fdaeb65-kube-api-access-wdbjt\") pod \"seaweedfs-64568bcd49-qvr5s\" (UID: \"47088f3b-a2d1-4d78-bb79-7c969fdaeb65\") " pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:04.096851 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.096833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hd4\" (UniqueName: \"kubernetes.io/projected/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-kube-api-access-b2hd4\") pod \"kserve-controller-manager-844cf74686-qq7rx\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") " pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:04.173852 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.173764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:04.221651 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.221624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:04.312614 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.312584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-qq7rx"]
Jul 29 22:56:04.316231 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:56:04.316198    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309355c0_fbc5_42b4_bdaf_9bf04a50e2f3.slice/crio-72ebdf4f33c2e428e30e545cc824571902e4d8661cd9397faa486360322b4891 WatchSource:0}: Error finding container 72ebdf4f33c2e428e30e545cc824571902e4d8661cd9397faa486360322b4891: Status 404 returned error can't find the container with id 72ebdf4f33c2e428e30e545cc824571902e4d8661cd9397faa486360322b4891
Jul 29 22:56:04.358528 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.358505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qvr5s"]
Jul 29 22:56:04.361150 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:56:04.361121    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47088f3b_a2d1_4d78_bb79_7c969fdaeb65.slice/crio-b05f406c0a3756cf917b67c3607a185ad95e6e5796169a54a3587954bf99a047 WatchSource:0}: Error finding container b05f406c0a3756cf917b67c3607a185ad95e6e5796169a54a3587954bf99a047: Status 404 returned error can't find the container with id b05f406c0a3756cf917b67c3607a185ad95e6e5796169a54a3587954bf99a047
Jul 29 22:56:04.594527 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.594488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qvr5s" event={"ID":"47088f3b-a2d1-4d78-bb79-7c969fdaeb65","Type":"ContainerStarted","Data":"b05f406c0a3756cf917b67c3607a185ad95e6e5796169a54a3587954bf99a047"}
Jul 29 22:56:04.595651 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:04.595628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844cf74686-qq7rx" event={"ID":"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3","Type":"ContainerStarted","Data":"72ebdf4f33c2e428e30e545cc824571902e4d8661cd9397faa486360322b4891"}
Jul 29 22:56:08.617224 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:08.617189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qvr5s" event={"ID":"47088f3b-a2d1-4d78-bb79-7c969fdaeb65","Type":"ContainerStarted","Data":"ba9ead08c0f07ecd693bd9f1b9d1f81bb187a85ef13d22274acb12865f492bd5"}
Jul 29 22:56:08.617691 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:08.617298    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:08.618683 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:08.618660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844cf74686-qq7rx" event={"ID":"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3","Type":"ContainerStarted","Data":"269680a2dac791a772c37ad67fd47903eced8cc33d16b8ed471067ca15df6367"}
Jul 29 22:56:08.618825 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:08.618807    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:08.633989 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:08.633935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qvr5s" podStartSLOduration=1.923887004 podStartE2EDuration="5.633916506s" totalImagesPullingTime="3.710029502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:56:08.632679254 +0000 UTC m=+584.541369622" watchObservedRunningTime="2026-07-29 22:56:08.633916506 +0000 UTC m=+584.542606866"
Jul 29 22:56:08.649504 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:08.649446    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-844cf74686-qq7rx" podStartSLOduration=1.9655431239999999 podStartE2EDuration="5.649426831s" totalImagesPullingTime="3.683883707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:56:08.648386771 +0000 UTC m=+584.557077141" watchObservedRunningTime="2026-07-29 22:56:08.649426831 +0000 UTC m=+584.558117189"
Jul 29 22:56:14.625626 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:14.625598    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qvr5s"
Jul 29 22:56:39.627776 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:39.627744    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:56:45.953344 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:45.953310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"]
Jul 29 22:56:45.958799 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:45.958763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:45.961279 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:45.961255    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 22:56:45.961279 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:45.961276    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 22:56:45.961682 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:45.961665    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n86fd\""
Jul 29 22:56:45.966010 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:45.965985    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"]
Jul 29 22:56:46.030286 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.030245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdkv\" (UniqueName: \"kubernetes.io/projected/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-kube-api-access-xtdkv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.030453 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.030307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.030453 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.030384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.131445 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.131411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.131645 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.131466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.131645 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.131498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdkv\" (UniqueName: \"kubernetes.io/projected/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-kube-api-access-xtdkv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.131920 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.131896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.132005 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.131898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.140836 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.140806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdkv\" (UniqueName: \"kubernetes.io/projected/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-kube-api-access-xtdkv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.270756 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.270656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:46.403444 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.403407    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"]
Jul 29 22:56:46.410093 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:56:46.409947    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca0bad3_4a2b_4f76_bc5f_7e803ddbbb9f.slice/crio-307f51f580f28647da68b2adab640a89cc3a7d038a97d2995ebe43a238d556a6 WatchSource:0}: Error finding container 307f51f580f28647da68b2adab640a89cc3a7d038a97d2995ebe43a238d556a6: Status 404 returned error can't find the container with id 307f51f580f28647da68b2adab640a89cc3a7d038a97d2995ebe43a238d556a6
Jul 29 22:56:46.782253 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.782220    2580 generic.go:356] "Generic (PLEG): container finished" podID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerID="70be663deefbc89fbb6e01a28a06097c844a5e259f0124ec2a57f2ef2c737c2f" exitCode=0
Jul 29 22:56:46.782433 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.782300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs" event={"ID":"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f","Type":"ContainerDied","Data":"70be663deefbc89fbb6e01a28a06097c844a5e259f0124ec2a57f2ef2c737c2f"}
Jul 29 22:56:46.782433 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.782343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs" event={"ID":"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f","Type":"ContainerStarted","Data":"307f51f580f28647da68b2adab640a89cc3a7d038a97d2995ebe43a238d556a6"}
Jul 29 22:56:46.783263 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:46.783248    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:56:49.800451 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:49.800411    2580 generic.go:356] "Generic (PLEG): container finished" podID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerID="21ca5cca051d9983dc9c074e272178e8684584d3bfe2ea10c71484fffc60eb7d" exitCode=0
Jul 29 22:56:49.800920 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:49.800498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs" event={"ID":"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f","Type":"ContainerDied","Data":"21ca5cca051d9983dc9c074e272178e8684584d3bfe2ea10c71484fffc60eb7d"}
Jul 29 22:56:50.806249 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:50.806214    2580 generic.go:356] "Generic (PLEG): container finished" podID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerID="3f84c8544a0cbc1eadef5d443a4ce766cd87188a2d7b3131cd5356a3e80d45c1" exitCode=0
Jul 29 22:56:50.806617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:50.806310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs" event={"ID":"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f","Type":"ContainerDied","Data":"3f84c8544a0cbc1eadef5d443a4ce766cd87188a2d7b3131cd5356a3e80d45c1"}
Jul 29 22:56:51.740822 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.740772    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54bbdcdd48-chw9c"]
Jul 29 22:56:51.743118 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.743095    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.756336 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.756312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54bbdcdd48-chw9c"]
Jul 29 22:56:51.781210 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781172    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hsx\" (UniqueName: \"kubernetes.io/projected/eddb29a1-f303-49a6-8836-6022d7dc38fa-kube-api-access-f8hsx\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.781210 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-serving-cert\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.781424 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-trusted-ca-bundle\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.781424 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-oauth-serving-cert\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.781424 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-oauth-config\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.781424 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-config\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.781424 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.781362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-service-ca\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-service-ca\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hsx\" (UniqueName: \"kubernetes.io/projected/eddb29a1-f303-49a6-8836-6022d7dc38fa-kube-api-access-f8hsx\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-serving-cert\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-trusted-ca-bundle\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-oauth-serving-cert\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-oauth-config\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.882702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.882474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-config\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.883162 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.883001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-service-ca\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.883162 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.883018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-oauth-serving-cert\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.883248 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.883197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-config\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.883304 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.883283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eddb29a1-f303-49a6-8836-6022d7dc38fa-trusted-ca-bundle\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.885717 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.885691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-oauth-config\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.885717 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.885709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eddb29a1-f303-49a6-8836-6022d7dc38fa-console-serving-cert\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.890593 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.890567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hsx\" (UniqueName: \"kubernetes.io/projected/eddb29a1-f303-49a6-8836-6022d7dc38fa-kube-api-access-f8hsx\") pod \"console-54bbdcdd48-chw9c\" (UID: \"eddb29a1-f303-49a6-8836-6022d7dc38fa\") " pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:51.938774 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.938743    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:51.983002 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.982975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-bundle\") pod \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") "
Jul 29 22:56:51.983174 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.983018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-util\") pod \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") "
Jul 29 22:56:51.983174 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.983102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtdkv\" (UniqueName: \"kubernetes.io/projected/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-kube-api-access-xtdkv\") pod \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\" (UID: \"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f\") "
Jul 29 22:56:51.983603 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.983581    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-bundle" (OuterVolumeSpecName: "bundle") pod "eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" (UID: "eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:56:51.985196 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.985166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-kube-api-access-xtdkv" (OuterVolumeSpecName: "kube-api-access-xtdkv") pod "eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" (UID: "eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f"). InnerVolumeSpecName "kube-api-access-xtdkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:56:51.988130 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:51.988093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-util" (OuterVolumeSpecName: "util") pod "eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" (UID: "eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:56:52.053437 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.053397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:56:52.083852 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.083821    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:56:52.083852 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.083848    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:56:52.083852 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.083857    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdkv\" (UniqueName: \"kubernetes.io/projected/eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f-kube-api-access-xtdkv\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:56:52.178755 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.178720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54bbdcdd48-chw9c"]
Jul 29 22:56:52.181708 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:56:52.181673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddb29a1_f303_49a6_8836_6022d7dc38fa.slice/crio-1a486ecbe05d7707f6254d570797472bcf24da436a525f7b0dede04508365981 WatchSource:0}: Error finding container 1a486ecbe05d7707f6254d570797472bcf24da436a525f7b0dede04508365981: Status 404 returned error can't find the container with id 1a486ecbe05d7707f6254d570797472bcf24da436a525f7b0dede04508365981
Jul 29 22:56:52.817932 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.817901    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs"
Jul 29 22:56:52.818109 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.817901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754022nqs" event={"ID":"eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f","Type":"ContainerDied","Data":"307f51f580f28647da68b2adab640a89cc3a7d038a97d2995ebe43a238d556a6"}
Jul 29 22:56:52.818109 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.818035    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307f51f580f28647da68b2adab640a89cc3a7d038a97d2995ebe43a238d556a6"
Jul 29 22:56:52.819488 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.819461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bbdcdd48-chw9c" event={"ID":"eddb29a1-f303-49a6-8836-6022d7dc38fa","Type":"ContainerStarted","Data":"e7702fb7723fe1080dc75753bad45f942ab57f2fee68fc635970779a9517f84d"}
Jul 29 22:56:52.819615 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.819504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54bbdcdd48-chw9c" event={"ID":"eddb29a1-f303-49a6-8836-6022d7dc38fa","Type":"ContainerStarted","Data":"1a486ecbe05d7707f6254d570797472bcf24da436a525f7b0dede04508365981"}
Jul 29 22:56:52.836287 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:52.836176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54bbdcdd48-chw9c" podStartSLOduration=1.836160691 podStartE2EDuration="1.836160691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:56:52.834933265 +0000 UTC m=+628.743623625" watchObservedRunningTime="2026-07-29 22:56:52.836160691 +0000 UTC m=+628.744851093"
Jul 29 22:56:58.820067 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820032    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"]
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820362    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="util"
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820376    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="util"
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820393    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="extract"
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820399    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="extract"
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820409    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="pull"
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820415    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="pull"
Jul 29 22:56:58.820507 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.820468    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca0bad3-4a2b-4f76-bc5f-7e803ddbbb9f" containerName="extract"
Jul 29 22:56:58.823617 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.823601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:58.825969 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.825947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 29 22:56:58.826096 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.826035    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 29 22:56:58.826173 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.826159    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7vz9s\""
Jul 29 22:56:58.840194 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.840175    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"]
Jul 29 22:56:58.943826 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.943770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlzx\" (UniqueName: \"kubernetes.io/projected/4cc34551-f61d-4259-901f-7076e9e3c151-kube-api-access-djlzx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-rtdt2\" (UID: \"4cc34551-f61d-4259-901f-7076e9e3c151\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:58.943826 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:58.943833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cc34551-f61d-4259-901f-7076e9e3c151-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-rtdt2\" (UID: \"4cc34551-f61d-4259-901f-7076e9e3c151\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:59.045008 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.044962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlzx\" (UniqueName: \"kubernetes.io/projected/4cc34551-f61d-4259-901f-7076e9e3c151-kube-api-access-djlzx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-rtdt2\" (UID: \"4cc34551-f61d-4259-901f-7076e9e3c151\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:59.045008 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.045011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cc34551-f61d-4259-901f-7076e9e3c151-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-rtdt2\" (UID: \"4cc34551-f61d-4259-901f-7076e9e3c151\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:59.045398 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.045377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cc34551-f61d-4259-901f-7076e9e3c151-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-rtdt2\" (UID: \"4cc34551-f61d-4259-901f-7076e9e3c151\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:59.053361 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.053337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlzx\" (UniqueName: \"kubernetes.io/projected/4cc34551-f61d-4259-901f-7076e9e3c151-kube-api-access-djlzx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-rtdt2\" (UID: \"4cc34551-f61d-4259-901f-7076e9e3c151\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:59.132400 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.132302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"
Jul 29 22:56:59.263231 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.263204    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2"]
Jul 29 22:56:59.266114 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:56:59.266086    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc34551_f61d_4259_901f_7076e9e3c151.slice/crio-86e46014a925037d57c54ced7caf8dda63e58443ba27973eae8e3f6be5e87895 WatchSource:0}: Error finding container 86e46014a925037d57c54ced7caf8dda63e58443ba27973eae8e3f6be5e87895: Status 404 returned error can't find the container with id 86e46014a925037d57c54ced7caf8dda63e58443ba27973eae8e3f6be5e87895
Jul 29 22:56:59.854112 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:56:59.854081    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2" event={"ID":"4cc34551-f61d-4259-901f-7076e9e3c151","Type":"ContainerStarted","Data":"86e46014a925037d57c54ced7caf8dda63e58443ba27973eae8e3f6be5e87895"}
Jul 29 22:57:01.865735 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:01.865694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2" event={"ID":"4cc34551-f61d-4259-901f-7076e9e3c151","Type":"ContainerStarted","Data":"e33566784879bcbc6dbc4deb1503bb9a7b16719725dc63ab18acbc666a40e125"}
Jul 29 22:57:01.885889 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:01.885826    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-rtdt2" podStartSLOduration=1.8220125010000001 podStartE2EDuration="3.885807472s" totalImagesPullingTime="2.063794971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:01.884913949 +0000 UTC m=+637.793604308" watchObservedRunningTime="2026-07-29 22:57:01.885807472 +0000 UTC m=+637.794497819"
Jul 29 22:57:02.053890 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:02.053850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:57:02.054202 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:02.053913    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:57:02.059197 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:02.059170    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:57:02.873663 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:02.873639    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54bbdcdd48-chw9c"
Jul 29 22:57:02.934249 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:02.934219    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5985c6b55-g45lj"]
Jul 29 22:57:07.121673 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.121636    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"]
Jul 29 22:57:07.125455 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.125437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.127892 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.127869    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n86fd\""
Jul 29 22:57:07.128031 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.127922    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 22:57:07.128475 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.128457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 22:57:07.134842 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.134820    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"]
Jul 29 22:57:07.217260 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.217223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.217448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.217275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.217448 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.217298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwh7\" (UniqueName: \"kubernetes.io/projected/99d44a49-80fc-4890-b9f5-54bff4c0642b-kube-api-access-6rwh7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.318143 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.318103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwh7\" (UniqueName: \"kubernetes.io/projected/99d44a49-80fc-4890-b9f5-54bff4c0642b-kube-api-access-6rwh7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.318345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.318195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.318345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.318237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.318581 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.318562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.318655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.318609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.344056 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.344024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwh7\" (UniqueName: \"kubernetes.io/projected/99d44a49-80fc-4890-b9f5-54bff4c0642b-kube-api-access-6rwh7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.435513 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.435416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:07.566153 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.566125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"]
Jul 29 22:57:07.567882 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:07.567852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d44a49_80fc_4890_b9f5_54bff4c0642b.slice/crio-6a23a21da61a2e4d834c66a021758e13ec6d79bbec378b04080eb344a9b435d1 WatchSource:0}: Error finding container 6a23a21da61a2e4d834c66a021758e13ec6d79bbec378b04080eb344a9b435d1: Status 404 returned error can't find the container with id 6a23a21da61a2e4d834c66a021758e13ec6d79bbec378b04080eb344a9b435d1
Jul 29 22:57:07.892872 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.892836    2580 generic.go:356] "Generic (PLEG): container finished" podID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerID="44f64cf86410f075fc3fda7bcde5d48b7075f1411487719d5dbaa6ad52268b9e" exitCode=0
Jul 29 22:57:07.893079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.892920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw" event={"ID":"99d44a49-80fc-4890-b9f5-54bff4c0642b","Type":"ContainerDied","Data":"44f64cf86410f075fc3fda7bcde5d48b7075f1411487719d5dbaa6ad52268b9e"}
Jul 29 22:57:07.893079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:07.892968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw" event={"ID":"99d44a49-80fc-4890-b9f5-54bff4c0642b","Type":"ContainerStarted","Data":"6a23a21da61a2e4d834c66a021758e13ec6d79bbec378b04080eb344a9b435d1"}
Jul 29 22:57:10.910707 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:10.910673    2580 generic.go:356] "Generic (PLEG): container finished" podID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerID="0e13562752e077aa2d3de3ac8db5ab884521587aec66160396e5ce7498e44e2a" exitCode=0
Jul 29 22:57:10.911134 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:10.910763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw" event={"ID":"99d44a49-80fc-4890-b9f5-54bff4c0642b","Type":"ContainerDied","Data":"0e13562752e077aa2d3de3ac8db5ab884521587aec66160396e5ce7498e44e2a"}
Jul 29 22:57:11.916990 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:11.916954    2580 generic.go:356] "Generic (PLEG): container finished" podID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerID="94c8f0ee8d26eb850a3a36dc7bcf0be0f61fa8446579812eacd86b1ddbe26a43" exitCode=0
Jul 29 22:57:11.917395 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:11.917030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw" event={"ID":"99d44a49-80fc-4890-b9f5-54bff4c0642b","Type":"ContainerDied","Data":"94c8f0ee8d26eb850a3a36dc7bcf0be0f61fa8446579812eacd86b1ddbe26a43"}
Jul 29 22:57:13.049698 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.049676    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:13.172839 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.172710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-util\") pod \"99d44a49-80fc-4890-b9f5-54bff4c0642b\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") "
Jul 29 22:57:13.172839 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.172839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rwh7\" (UniqueName: \"kubernetes.io/projected/99d44a49-80fc-4890-b9f5-54bff4c0642b-kube-api-access-6rwh7\") pod \"99d44a49-80fc-4890-b9f5-54bff4c0642b\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") "
Jul 29 22:57:13.173079 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.172903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-bundle\") pod \"99d44a49-80fc-4890-b9f5-54bff4c0642b\" (UID: \"99d44a49-80fc-4890-b9f5-54bff4c0642b\") "
Jul 29 22:57:13.173246 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.173209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-bundle" (OuterVolumeSpecName: "bundle") pod "99d44a49-80fc-4890-b9f5-54bff4c0642b" (UID: "99d44a49-80fc-4890-b9f5-54bff4c0642b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:13.174939 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.174906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d44a49-80fc-4890-b9f5-54bff4c0642b-kube-api-access-6rwh7" (OuterVolumeSpecName: "kube-api-access-6rwh7") pod "99d44a49-80fc-4890-b9f5-54bff4c0642b" (UID: "99d44a49-80fc-4890-b9f5-54bff4c0642b"). InnerVolumeSpecName "kube-api-access-6rwh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:57:13.177406 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.177371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-util" (OuterVolumeSpecName: "util") pod "99d44a49-80fc-4890-b9f5-54bff4c0642b" (UID: "99d44a49-80fc-4890-b9f5-54bff4c0642b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:13.274511 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.274477    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rwh7\" (UniqueName: \"kubernetes.io/projected/99d44a49-80fc-4890-b9f5-54bff4c0642b-kube-api-access-6rwh7\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:13.274511 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.274505    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:13.274511 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.274514    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99d44a49-80fc-4890-b9f5-54bff4c0642b-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:13.928310 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.928273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw" event={"ID":"99d44a49-80fc-4890-b9f5-54bff4c0642b","Type":"ContainerDied","Data":"6a23a21da61a2e4d834c66a021758e13ec6d79bbec378b04080eb344a9b435d1"}
Jul 29 22:57:13.928310 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.928296    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg99pw"
Jul 29 22:57:13.928489 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:13.928328    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a23a21da61a2e4d834c66a021758e13ec6d79bbec378b04080eb344a9b435d1"
Jul 29 22:57:27.955467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:27.955424    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5985c6b55-g45lj" podUID="b90c6aa2-a7a7-4785-a399-23028bc39cad" containerName="console" containerID="cri-o://e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01" gracePeriod=15
Jul 29 22:57:28.202734 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.202711    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:57:28.297716 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-service-ca\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.297716 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-oauth-serving-cert\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.297716 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-trusted-ca-bundle\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.297716 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-oauth-config\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.298080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297730    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx5cv\" (UniqueName: \"kubernetes.io/projected/b90c6aa2-a7a7-4785-a399-23028bc39cad-kube-api-access-xx5cv\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.298080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-config\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.298080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.297898    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-serving-cert\") pod \"b90c6aa2-a7a7-4785-a399-23028bc39cad\" (UID: \"b90c6aa2-a7a7-4785-a399-23028bc39cad\") "
Jul 29 22:57:28.298255 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.298115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-service-ca" (OuterVolumeSpecName: "service-ca") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:57:28.298318 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.298153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:57:28.298318 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.298266    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:57:28.298318 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.298305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-config" (OuterVolumeSpecName: "console-config") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:57:28.300003 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.299977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:57:28.300113 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.300007    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90c6aa2-a7a7-4785-a399-23028bc39cad-kube-api-access-xx5cv" (OuterVolumeSpecName: "kube-api-access-xx5cv") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "kube-api-access-xx5cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:57:28.300113 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.300019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b90c6aa2-a7a7-4785-a399-23028bc39cad" (UID: "b90c6aa2-a7a7-4785-a399-23028bc39cad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:57:28.398917 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398887    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-service-ca\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.398917 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398914    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-oauth-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.398917 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398923    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-trusted-ca-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.399131 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398932    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-oauth-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.399131 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398940    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5cv\" (UniqueName: \"kubernetes.io/projected/b90c6aa2-a7a7-4785-a399-23028bc39cad-kube-api-access-xx5cv\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.399131 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398948    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-config\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.399131 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.398958    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b90c6aa2-a7a7-4785-a399-23028bc39cad-console-serving-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:28.996608 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.996533    2580 generic.go:356] "Generic (PLEG): container finished" podID="b90c6aa2-a7a7-4785-a399-23028bc39cad" containerID="e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01" exitCode=0
Jul 29 22:57:28.996608 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.996579    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5985c6b55-g45lj"
Jul 29 22:57:28.997051 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.996620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5985c6b55-g45lj" event={"ID":"b90c6aa2-a7a7-4785-a399-23028bc39cad","Type":"ContainerDied","Data":"e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01"}
Jul 29 22:57:28.997051 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.996670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5985c6b55-g45lj" event={"ID":"b90c6aa2-a7a7-4785-a399-23028bc39cad","Type":"ContainerDied","Data":"79f039cac8240bb2ae0c4120cc92f5b7534e5f3b7bbacea614b0525c64377e0f"}
Jul 29 22:57:28.997051 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:28.996691    2580 scope.go:117] "RemoveContainer" containerID="e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01"
Jul 29 22:57:29.005128 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:29.005112    2580 scope.go:117] "RemoveContainer" containerID="e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01"
Jul 29 22:57:29.005378 ip-10-0-131-193 kubenswrapper[2580]: E0729 22:57:29.005360    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01\": container with ID starting with e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01 not found: ID does not exist" containerID="e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01"
Jul 29 22:57:29.005422 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:29.005388    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01"} err="failed to get container status \"e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01\": rpc error: code = NotFound desc = could not find container \"e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01\": container with ID starting with e93cc7a8fdf9715937d0b7b9b74a0abba27cfc9c66d01454a23ccc2dea91ba01 not found: ID does not exist"
Jul 29 22:57:29.011150 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:29.011123    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5985c6b55-g45lj"]
Jul 29 22:57:29.013581 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:29.013560    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5985c6b55-g45lj"]
Jul 29 22:57:30.569307 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:30.569273    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b90c6aa2-a7a7-4785-a399-23028bc39cad" path="/var/lib/kubelet/pods/b90c6aa2-a7a7-4785-a399-23028bc39cad/volumes"
Jul 29 22:57:35.625708 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.625671    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"]
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.625996    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="extract"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="extract"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626021    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="util"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626026    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="util"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626033    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="pull"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="pull"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626056    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90c6aa2-a7a7-4785-a399-23028bc39cad" containerName="console"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626061    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90c6aa2-a7a7-4785-a399-23028bc39cad" containerName="console"
Jul 29 22:57:35.626114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626111    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b90c6aa2-a7a7-4785-a399-23028bc39cad" containerName="console"
Jul 29 22:57:35.626394 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.626121    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d44a49-80fc-4890-b9f5-54bff4c0642b" containerName="extract"
Jul 29 22:57:35.630643 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.630627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.632962 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.632942    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 29 22:57:35.633041 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.632977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 29 22:57:35.633041 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.632988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kr97z\""
Jul 29 22:57:35.633227 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.633208    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 29 22:57:35.633330 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.633315    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 22:57:35.643754 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.643730    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"]
Jul 29 22:57:35.755519 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd945e8-9126-4f9a-939b-2b2844bb8443-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.755729 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.755729 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.755729 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.755729 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.755729 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmhc\" (UniqueName: \"kubernetes.io/projected/bbd945e8-9126-4f9a-939b-2b2844bb8443-kube-api-access-dzmhc\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.755947 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.755750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.856926 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.856880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.856926 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.856929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.857182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.856962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.857182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.856992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.857182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.857017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmhc\" (UniqueName: \"kubernetes.io/projected/bbd945e8-9126-4f9a-939b-2b2844bb8443-kube-api-access-dzmhc\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.857182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.857041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.857182 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.857088    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd945e8-9126-4f9a-939b-2b2844bb8443-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.857737 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.857711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.859379 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.859353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.859476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.859394    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd945e8-9126-4f9a-939b-2b2844bb8443-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.859476 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.859464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.859600 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.859496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.864777 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.864748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmhc\" (UniqueName: \"kubernetes.io/projected/bbd945e8-9126-4f9a-939b-2b2844bb8443-kube-api-access-dzmhc\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.864897 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.864807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd945e8-9126-4f9a-939b-2b2844bb8443-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dz27l\" (UID: \"bbd945e8-9126-4f9a-939b-2b2844bb8443\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:35.939324 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:35.939220    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:36.074702 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:36.074675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"]
Jul 29 22:57:36.077293 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:36.077262    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd945e8_9126_4f9a_939b_2b2844bb8443.slice/crio-59398bd60e85ee4ff438c5c0a932f1cd4ddd83e3cd67853ac1a012f1221f01c5 WatchSource:0}: Error finding container 59398bd60e85ee4ff438c5c0a932f1cd4ddd83e3cd67853ac1a012f1221f01c5: Status 404 returned error can't find the container with id 59398bd60e85ee4ff438c5c0a932f1cd4ddd83e3cd67853ac1a012f1221f01c5
Jul 29 22:57:37.035219 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:37.035185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l" event={"ID":"bbd945e8-9126-4f9a-939b-2b2844bb8443","Type":"ContainerStarted","Data":"59398bd60e85ee4ff438c5c0a932f1cd4ddd83e3cd67853ac1a012f1221f01c5"}
Jul 29 22:57:38.995662 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:38.995625    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 22:57:38.996009 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:38.995691    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 22:57:40.050485 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:40.050450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l" event={"ID":"bbd945e8-9126-4f9a-939b-2b2844bb8443","Type":"ContainerStarted","Data":"ac15e090a261e2e078c10d3a20bcd062dd67544f41ad033aedaa750c59ebb47e"}
Jul 29 22:57:40.051087 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:40.050717    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:40.052858 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:40.052830    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-dz27l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 22:57:40.053006 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:40.052877    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l" podUID="bbd945e8-9126-4f9a-939b-2b2844bb8443" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 22:57:40.072494 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:40.072445    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l" podStartSLOduration=2.156499514 podStartE2EDuration="5.072428907s" totalImagesPullingTime="2.915929393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:40.070713861 +0000 UTC m=+675.979404220" watchObservedRunningTime="2026-07-29 22:57:40.072428907 +0000 UTC m=+675.981119265"
Jul 29 22:57:41.056247 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:41.056222    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dz27l"
Jul 29 22:57:42.028807 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.028645    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"]
Jul 29 22:57:42.033029 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.033006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.035190 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.035168    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-rs52k\""
Jul 29 22:57:42.043017 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.042993    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"]
Jul 29 22:57:42.114913 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.114875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rq5\" (UniqueName: \"kubernetes.io/projected/6e724f3c-88f2-4fcc-924e-95054f341fd2-kube-api-access-r6rq5\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.114927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.114954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.114977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e724f3c-88f2-4fcc-924e-95054f341fd2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.115040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.115117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.115164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.115181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.115373 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.115202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216125 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e724f3c-88f2-4fcc-924e-95054f341fd2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216316 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216578 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216634 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rq5\" (UniqueName: \"kubernetes.io/projected/6e724f3c-88f2-4fcc-924e-95054f341fd2-kube-api-access-r6rq5\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216688 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216688 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.216818 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.216699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.217192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.217020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.217192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.217115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e724f3c-88f2-4fcc-924e-95054f341fd2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.217192 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.217151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.218467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.218446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.218728 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.218712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.223655 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.223635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e724f3c-88f2-4fcc-924e-95054f341fd2-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.224071 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.224052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rq5\" (UniqueName: \"kubernetes.io/projected/6e724f3c-88f2-4fcc-924e-95054f341fd2-kube-api-access-r6rq5\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw\" (UID: \"6e724f3c-88f2-4fcc-924e-95054f341fd2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.345277 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.345234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:42.478294 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:42.478247    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"]
Jul 29 22:57:42.481588 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:42.481559    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e724f3c_88f2_4fcc_924e_95054f341fd2.slice/crio-f7bfc12da4e68fecdab6bf93ef354618cc76153068579237def762118a8eb27a WatchSource:0}: Error finding container f7bfc12da4e68fecdab6bf93ef354618cc76153068579237def762118a8eb27a: Status 404 returned error can't find the container with id f7bfc12da4e68fecdab6bf93ef354618cc76153068579237def762118a8eb27a
Jul 29 22:57:43.071900 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:43.071859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" event={"ID":"6e724f3c-88f2-4fcc-924e-95054f341fd2","Type":"ContainerStarted","Data":"f7bfc12da4e68fecdab6bf93ef354618cc76153068579237def762118a8eb27a"}
Jul 29 22:57:45.550569 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:45.550531    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 22:57:45.550973 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:45.550619    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 22:57:45.550973 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:45.550668    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 22:57:46.088904 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:46.088870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" event={"ID":"6e724f3c-88f2-4fcc-924e-95054f341fd2","Type":"ContainerStarted","Data":"659fde2e5fe70ee279988edd6e2798fd625d2bf00d7d50bfbef3e81040a5100a"}
Jul 29 22:57:46.107844 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:46.107779    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podStartSLOduration=1.041555062 podStartE2EDuration="4.107763048s" totalImagesPullingTime="3.066207986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:57:46.105121201 +0000 UTC m=+682.013811558" watchObservedRunningTime="2026-07-29 22:57:46.107763048 +0000 UTC m=+682.016453405"
Jul 29 22:57:46.346191 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:46.346104    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:46.350848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:46.350825    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:47.095208 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:47.095173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:47.098927 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:47.098897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 22:57:50.732108 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.732072    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"]
Jul 29 22:57:50.735639 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.735617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.737750 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.737726    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 22:57:50.737847 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.737766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 22:57:50.738239 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.738225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n86fd\""
Jul 29 22:57:50.743095 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.742934    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"]
Jul 29 22:57:50.825899 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.825861    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"]
Jul 29 22:57:50.829252 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.829237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:50.836672 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.836648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"]
Jul 29 22:57:50.894304 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.894264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.894467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.894315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.894467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.894424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4tx\" (UniqueName: \"kubernetes.io/projected/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-kube-api-access-hf4tx\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.929757 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.929725    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"]
Jul 29 22:57:50.933579 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.933563    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:50.941063 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.941038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"]
Jul 29 22:57:50.995886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.995772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.995886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.995835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.995886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.995879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4tx\" (UniqueName: \"kubernetes.io/projected/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-kube-api-access-hf4tx\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.996168 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.995908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:50.996168 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.995936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5rp\" (UniqueName: \"kubernetes.io/projected/ebd0956d-1ec7-481b-949e-094805ab5417-kube-api-access-4m5rp\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:50.996168 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.995967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:50.996314 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.996196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:50.996314 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:50.996258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:51.003698 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.003673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4tx\" (UniqueName: \"kubernetes.io/projected/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-kube-api-access-hf4tx\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:51.027044 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.027018    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"]
Jul 29 22:57:51.030532 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.030518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.037914 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.037769    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"]
Jul 29 22:57:51.045826 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.045799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:51.096544 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.096489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.096807 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.096693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.097648 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.096996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjfk\" (UniqueName: \"kubernetes.io/projected/d86dd113-ab93-4569-ab79-88b1535325eb-kube-api-access-5kjfk\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.097648 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.097065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5rp\" (UniqueName: \"kubernetes.io/projected/ebd0956d-1ec7-481b-949e-094805ab5417-kube-api-access-4m5rp\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.097648 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.097109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.097648 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.097153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.097648 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.097216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.097648 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.097608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.104363 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.104334    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5rp\" (UniqueName: \"kubernetes.io/projected/ebd0956d-1ec7-481b-949e-094805ab5417-kube-api-access-4m5rp\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.139094 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.139066    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:51.184195 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.184171    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"]
Jul 29 22:57:51.185834 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:51.185780    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cc6a4b_1401_4fbe_a7bd_238cc99f600c.slice/crio-996f712602c14a634e89d1fa665963dfca801640eb33c99a528719c2fa94fe29 WatchSource:0}: Error finding container 996f712602c14a634e89d1fa665963dfca801640eb33c99a528719c2fa94fe29: Status 404 returned error can't find the container with id 996f712602c14a634e89d1fa665963dfca801640eb33c99a528719c2fa94fe29
Jul 29 22:57:51.197984 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.197953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjfk\" (UniqueName: \"kubernetes.io/projected/d86dd113-ab93-4569-ab79-88b1535325eb-kube-api-access-5kjfk\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.198166 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.198166 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.198166 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198102    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8d8c\" (UniqueName: \"kubernetes.io/projected/5892656f-322d-4336-bfe9-f4a94d3aaf6d-kube-api-access-h8d8c\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.198166 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.198401 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.198588 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.198645 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.198603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.207772 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.207748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjfk\" (UniqueName: \"kubernetes.io/projected/d86dd113-ab93-4569-ab79-88b1535325eb-kube-api-access-5kjfk\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.243862 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.243774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:51.282319 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.282290    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"]
Jul 29 22:57:51.284898 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:51.284850    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd0956d_1ec7_481b_949e_094805ab5417.slice/crio-c8ceeeb785a8442778f0f254b43d3af2d02a774923227c1accc8e94fe1cbc524 WatchSource:0}: Error finding container c8ceeeb785a8442778f0f254b43d3af2d02a774923227c1accc8e94fe1cbc524: Status 404 returned error can't find the container with id c8ceeeb785a8442778f0f254b43d3af2d02a774923227c1accc8e94fe1cbc524
Jul 29 22:57:51.299143 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.298996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8d8c\" (UniqueName: \"kubernetes.io/projected/5892656f-322d-4336-bfe9-f4a94d3aaf6d-kube-api-access-h8d8c\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.300139 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.299172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.300139 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.299841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.300139 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.299866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.300139 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.300133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.306959 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.306941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8d8c\" (UniqueName: \"kubernetes.io/projected/5892656f-322d-4336-bfe9-f4a94d3aaf6d-kube-api-access-h8d8c\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.341076 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.341042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:51.386836 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.386747    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"]
Jul 29 22:57:51.387057 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:51.387032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86dd113_ab93_4569_ab79_88b1535325eb.slice/crio-52dc51e7feac721230bbe5af20e5beb17f6dd34d4a938ea06462f25d4d6dcbab WatchSource:0}: Error finding container 52dc51e7feac721230bbe5af20e5beb17f6dd34d4a938ea06462f25d4d6dcbab: Status 404 returned error can't find the container with id 52dc51e7feac721230bbe5af20e5beb17f6dd34d4a938ea06462f25d4d6dcbab
Jul 29 22:57:51.490654 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:51.490625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"]
Jul 29 22:57:51.497757 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:57:51.497692    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5892656f_322d_4336_bfe9_f4a94d3aaf6d.slice/crio-eb5f0a139f14917c8052f84906b30fcfe30bedea942909e8899b797002a1f2a1 WatchSource:0}: Error finding container eb5f0a139f14917c8052f84906b30fcfe30bedea942909e8899b797002a1f2a1: Status 404 returned error can't find the container with id eb5f0a139f14917c8052f84906b30fcfe30bedea942909e8899b797002a1f2a1
Jul 29 22:57:52.119273 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.119241    2580 generic.go:356] "Generic (PLEG): container finished" podID="ebd0956d-1ec7-481b-949e-094805ab5417" containerID="0af02515491fd84869da63897a9f33545c09ff39c305ec329c3618f2ca86497e" exitCode=0
Jul 29 22:57:52.119736 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.119305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k" event={"ID":"ebd0956d-1ec7-481b-949e-094805ab5417","Type":"ContainerDied","Data":"0af02515491fd84869da63897a9f33545c09ff39c305ec329c3618f2ca86497e"}
Jul 29 22:57:52.119736 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.119347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k" event={"ID":"ebd0956d-1ec7-481b-949e-094805ab5417","Type":"ContainerStarted","Data":"c8ceeeb785a8442778f0f254b43d3af2d02a774923227c1accc8e94fe1cbc524"}
Jul 29 22:57:52.120932 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.120909    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerID="47702855f79b0a782b301d4d8e6407c54f27511cce29a9aac91afcdf4380ca3e" exitCode=0
Jul 29 22:57:52.121070 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.120996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k" event={"ID":"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c","Type":"ContainerDied","Data":"47702855f79b0a782b301d4d8e6407c54f27511cce29a9aac91afcdf4380ca3e"}
Jul 29 22:57:52.121070 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.121065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k" event={"ID":"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c","Type":"ContainerStarted","Data":"996f712602c14a634e89d1fa665963dfca801640eb33c99a528719c2fa94fe29"}
Jul 29 22:57:52.122431 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.122409    2580 generic.go:356] "Generic (PLEG): container finished" podID="d86dd113-ab93-4569-ab79-88b1535325eb" containerID="ee704e030ad0824c4044e68c5a0446922a276013962536cfb1ddd6ab6d187e77" exitCode=0
Jul 29 22:57:52.122503 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.122474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9" event={"ID":"d86dd113-ab93-4569-ab79-88b1535325eb","Type":"ContainerDied","Data":"ee704e030ad0824c4044e68c5a0446922a276013962536cfb1ddd6ab6d187e77"}
Jul 29 22:57:52.122558 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.122524    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9" event={"ID":"d86dd113-ab93-4569-ab79-88b1535325eb","Type":"ContainerStarted","Data":"52dc51e7feac721230bbe5af20e5beb17f6dd34d4a938ea06462f25d4d6dcbab"}
Jul 29 22:57:52.124282 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.124262    2580 generic.go:356] "Generic (PLEG): container finished" podID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerID="f6e173a30bfc93adaa175d99dc6f57d0dd47b8388edc054b6c1a8e43a3cb1240" exitCode=0
Jul 29 22:57:52.124369 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.124324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt" event={"ID":"5892656f-322d-4336-bfe9-f4a94d3aaf6d","Type":"ContainerDied","Data":"f6e173a30bfc93adaa175d99dc6f57d0dd47b8388edc054b6c1a8e43a3cb1240"}
Jul 29 22:57:52.124369 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:52.124359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt" event={"ID":"5892656f-322d-4336-bfe9-f4a94d3aaf6d","Type":"ContainerStarted","Data":"eb5f0a139f14917c8052f84906b30fcfe30bedea942909e8899b797002a1f2a1"}
Jul 29 22:57:53.131099 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:53.131025    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerID="44362c12efc0a503596d9ba6bee1a7c49e5ed89ba2d53c579bc7271374ee4d9c" exitCode=0
Jul 29 22:57:53.131510 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:53.131104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k" event={"ID":"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c","Type":"ContainerDied","Data":"44362c12efc0a503596d9ba6bee1a7c49e5ed89ba2d53c579bc7271374ee4d9c"}
Jul 29 22:57:54.138400 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.138368    2580 generic.go:356] "Generic (PLEG): container finished" podID="ebd0956d-1ec7-481b-949e-094805ab5417" containerID="97e27c2a2dd6d5777e804d94c67e2fe1b4d7f0266b41b8a39056448604b5e65d" exitCode=0
Jul 29 22:57:54.138883 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.138445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k" event={"ID":"ebd0956d-1ec7-481b-949e-094805ab5417","Type":"ContainerDied","Data":"97e27c2a2dd6d5777e804d94c67e2fe1b4d7f0266b41b8a39056448604b5e65d"}
Jul 29 22:57:54.140508 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.140486    2580 generic.go:356] "Generic (PLEG): container finished" podID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerID="59dcdcc651c5c34ed80a8ded91cc55048bcec59ee77a61e17d9d10977509cff0" exitCode=0
Jul 29 22:57:54.140623 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.140574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k" event={"ID":"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c","Type":"ContainerDied","Data":"59dcdcc651c5c34ed80a8ded91cc55048bcec59ee77a61e17d9d10977509cff0"}
Jul 29 22:57:54.142369 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.142345    2580 generic.go:356] "Generic (PLEG): container finished" podID="d86dd113-ab93-4569-ab79-88b1535325eb" containerID="79ed0477222013c4fa9214e9a8a9db05861d715f44dc3b64db4efaa24b29b215" exitCode=0
Jul 29 22:57:54.142473 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.142451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9" event={"ID":"d86dd113-ab93-4569-ab79-88b1535325eb","Type":"ContainerDied","Data":"79ed0477222013c4fa9214e9a8a9db05861d715f44dc3b64db4efaa24b29b215"}
Jul 29 22:57:54.144407 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.144388    2580 generic.go:356] "Generic (PLEG): container finished" podID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerID="1c4fcf584e49a60a07ce65b19f2c6a7f0dcc41d323a08316dd5a7244d5ab59cf" exitCode=0
Jul 29 22:57:54.144490 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:54.144468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt" event={"ID":"5892656f-322d-4336-bfe9-f4a94d3aaf6d","Type":"ContainerDied","Data":"1c4fcf584e49a60a07ce65b19f2c6a7f0dcc41d323a08316dd5a7244d5ab59cf"}
Jul 29 22:57:55.152415 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.152377    2580 generic.go:356] "Generic (PLEG): container finished" podID="ebd0956d-1ec7-481b-949e-094805ab5417" containerID="c47d033e775302b22713c176140b820734badd533a5558228a34dceca107ce45" exitCode=0
Jul 29 22:57:55.152897 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.152464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k" event={"ID":"ebd0956d-1ec7-481b-949e-094805ab5417","Type":"ContainerDied","Data":"c47d033e775302b22713c176140b820734badd533a5558228a34dceca107ce45"}
Jul 29 22:57:55.154338 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.154312    2580 generic.go:356] "Generic (PLEG): container finished" podID="d86dd113-ab93-4569-ab79-88b1535325eb" containerID="6c980ea9abc1af42b75ba64181dffcaefbb6dbdd1045975589404da740230352" exitCode=0
Jul 29 22:57:55.154450 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.154399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9" event={"ID":"d86dd113-ab93-4569-ab79-88b1535325eb","Type":"ContainerDied","Data":"6c980ea9abc1af42b75ba64181dffcaefbb6dbdd1045975589404da740230352"}
Jul 29 22:57:55.156371 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.156345    2580 generic.go:356] "Generic (PLEG): container finished" podID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerID="ac0169d0df1ec29b3c250ac609875e6b9b2a6324de08b34102f3030f6fdfe023" exitCode=0
Jul 29 22:57:55.156457 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.156406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt" event={"ID":"5892656f-322d-4336-bfe9-f4a94d3aaf6d","Type":"ContainerDied","Data":"ac0169d0df1ec29b3c250ac609875e6b9b2a6324de08b34102f3030f6fdfe023"}
Jul 29 22:57:55.291748 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.291722    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:55.436467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.436375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-util\") pod \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") "
Jul 29 22:57:55.436467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.436411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-bundle\") pod \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") "
Jul 29 22:57:55.436467 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.436448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf4tx\" (UniqueName: \"kubernetes.io/projected/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-kube-api-access-hf4tx\") pod \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\" (UID: \"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c\") "
Jul 29 22:57:55.436986 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.436961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-bundle" (OuterVolumeSpecName: "bundle") pod "e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" (UID: "e0cc6a4b-1401-4fbe-a7bd-238cc99f600c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:55.438527 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.438499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-kube-api-access-hf4tx" (OuterVolumeSpecName: "kube-api-access-hf4tx") pod "e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" (UID: "e0cc6a4b-1401-4fbe-a7bd-238cc99f600c"). InnerVolumeSpecName "kube-api-access-hf4tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:57:55.441598 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.441574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-util" (OuterVolumeSpecName: "util") pod "e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" (UID: "e0cc6a4b-1401-4fbe-a7bd-238cc99f600c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:55.537148 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.537118    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:55.537148 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.537142    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:55.537148 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:55.537151    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf4tx\" (UniqueName: \"kubernetes.io/projected/e0cc6a4b-1401-4fbe-a7bd-238cc99f600c-kube-api-access-hf4tx\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.163700 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.163672    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k"
Jul 29 22:57:56.163700 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.163686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsgr4k" event={"ID":"e0cc6a4b-1401-4fbe-a7bd-238cc99f600c","Type":"ContainerDied","Data":"996f712602c14a634e89d1fa665963dfca801640eb33c99a528719c2fa94fe29"}
Jul 29 22:57:56.164120 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.163730    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="996f712602c14a634e89d1fa665963dfca801640eb33c99a528719c2fa94fe29"
Jul 29 22:57:56.312006 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.311979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:56.363862 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.363837    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:56.366801 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.366768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:56.444855 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.444754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-util\") pod \"d86dd113-ab93-4569-ab79-88b1535325eb\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") "
Jul 29 22:57:56.444855 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.444847    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-util\") pod \"ebd0956d-1ec7-481b-949e-094805ab5417\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") "
Jul 29 22:57:56.445072 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.444875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-bundle\") pod \"ebd0956d-1ec7-481b-949e-094805ab5417\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") "
Jul 29 22:57:56.445072 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.444896    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kjfk\" (UniqueName: \"kubernetes.io/projected/d86dd113-ab93-4569-ab79-88b1535325eb-kube-api-access-5kjfk\") pod \"d86dd113-ab93-4569-ab79-88b1535325eb\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") "
Jul 29 22:57:56.445072 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.444925    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-bundle\") pod \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") "
Jul 29 22:57:56.445072 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.444982    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-bundle\") pod \"d86dd113-ab93-4569-ab79-88b1535325eb\" (UID: \"d86dd113-ab93-4569-ab79-88b1535325eb\") "
Jul 29 22:57:56.445072 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.445002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-util\") pod \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") "
Jul 29 22:57:56.445072 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.445046    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8d8c\" (UniqueName: \"kubernetes.io/projected/5892656f-322d-4336-bfe9-f4a94d3aaf6d-kube-api-access-h8d8c\") pod \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\" (UID: \"5892656f-322d-4336-bfe9-f4a94d3aaf6d\") "
Jul 29 22:57:56.445338 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.445086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m5rp\" (UniqueName: \"kubernetes.io/projected/ebd0956d-1ec7-481b-949e-094805ab5417-kube-api-access-4m5rp\") pod \"ebd0956d-1ec7-481b-949e-094805ab5417\" (UID: \"ebd0956d-1ec7-481b-949e-094805ab5417\") "
Jul 29 22:57:56.445517 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.445462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-bundle" (OuterVolumeSpecName: "bundle") pod "5892656f-322d-4336-bfe9-f4a94d3aaf6d" (UID: "5892656f-322d-4336-bfe9-f4a94d3aaf6d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:56.445649 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.445598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-bundle" (OuterVolumeSpecName: "bundle") pod "ebd0956d-1ec7-481b-949e-094805ab5417" (UID: "ebd0956d-1ec7-481b-949e-094805ab5417"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:56.445918 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.445875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-bundle" (OuterVolumeSpecName: "bundle") pod "d86dd113-ab93-4569-ab79-88b1535325eb" (UID: "d86dd113-ab93-4569-ab79-88b1535325eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:56.447580 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.447554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86dd113-ab93-4569-ab79-88b1535325eb-kube-api-access-5kjfk" (OuterVolumeSpecName: "kube-api-access-5kjfk") pod "d86dd113-ab93-4569-ab79-88b1535325eb" (UID: "d86dd113-ab93-4569-ab79-88b1535325eb"). InnerVolumeSpecName "kube-api-access-5kjfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:57:56.447954 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.447932    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd0956d-1ec7-481b-949e-094805ab5417-kube-api-access-4m5rp" (OuterVolumeSpecName: "kube-api-access-4m5rp") pod "ebd0956d-1ec7-481b-949e-094805ab5417" (UID: "ebd0956d-1ec7-481b-949e-094805ab5417"). InnerVolumeSpecName "kube-api-access-4m5rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:57:56.448117 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.448096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5892656f-322d-4336-bfe9-f4a94d3aaf6d-kube-api-access-h8d8c" (OuterVolumeSpecName: "kube-api-access-h8d8c") pod "5892656f-322d-4336-bfe9-f4a94d3aaf6d" (UID: "5892656f-322d-4336-bfe9-f4a94d3aaf6d"). InnerVolumeSpecName "kube-api-access-h8d8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:57:56.451758 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.451735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-util" (OuterVolumeSpecName: "util") pod "d86dd113-ab93-4569-ab79-88b1535325eb" (UID: "d86dd113-ab93-4569-ab79-88b1535325eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:56.452027 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.451993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-util" (OuterVolumeSpecName: "util") pod "ebd0956d-1ec7-481b-949e-094805ab5417" (UID: "ebd0956d-1ec7-481b-949e-094805ab5417"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:56.452093 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.452025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-util" (OuterVolumeSpecName: "util") pod "5892656f-322d-4336-bfe9-f4a94d3aaf6d" (UID: "5892656f-322d-4336-bfe9-f4a94d3aaf6d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:57:56.546164 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546120    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546164 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546159    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546187    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebd0956d-1ec7-481b-949e-094805ab5417-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546196    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjfk\" (UniqueName: \"kubernetes.io/projected/d86dd113-ab93-4569-ab79-88b1535325eb-kube-api-access-5kjfk\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546206    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546214    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d86dd113-ab93-4569-ab79-88b1535325eb-bundle\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546222    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5892656f-322d-4336-bfe9-f4a94d3aaf6d-util\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546230    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8d8c\" (UniqueName: \"kubernetes.io/projected/5892656f-322d-4336-bfe9-f4a94d3aaf6d-kube-api-access-h8d8c\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:56.546354 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:56.546238    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5rp\" (UniqueName: \"kubernetes.io/projected/ebd0956d-1ec7-481b-949e-094805ab5417-kube-api-access-4m5rp\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:57:57.170292 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.170256    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt"
Jul 29 22:57:57.170721 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.170297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0bnmtt" event={"ID":"5892656f-322d-4336-bfe9-f4a94d3aaf6d","Type":"ContainerDied","Data":"eb5f0a139f14917c8052f84906b30fcfe30bedea942909e8899b797002a1f2a1"}
Jul 29 22:57:57.170721 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.170349    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5f0a139f14917c8052f84906b30fcfe30bedea942909e8899b797002a1f2a1"
Jul 29 22:57:57.172343 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.172316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k"
Jul 29 22:57:57.172343 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.172324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpst7k" event={"ID":"ebd0956d-1ec7-481b-949e-094805ab5417","Type":"ContainerDied","Data":"c8ceeeb785a8442778f0f254b43d3af2d02a774923227c1accc8e94fe1cbc524"}
Jul 29 22:57:57.172517 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.172388    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ceeeb785a8442778f0f254b43d3af2d02a774923227c1accc8e94fe1cbc524"
Jul 29 22:57:57.174418 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.174381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9" event={"ID":"d86dd113-ab93-4569-ab79-88b1535325eb","Type":"ContainerDied","Data":"52dc51e7feac721230bbe5af20e5beb17f6dd34d4a938ea06462f25d4d6dcbab"}
Jul 29 22:57:57.174418 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.174406    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053bgrt9"
Jul 29 22:57:57.174418 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:57:57.174416    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52dc51e7feac721230bbe5af20e5beb17f6dd34d4a938ea06462f25d4d6dcbab"
Jul 29 22:58:05.789502 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789468    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-b9spq"]
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789805    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789816    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789823    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="extract"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="extract"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="util"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="util"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789862    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="extract"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789867    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="extract"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789874    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789879    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789885    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="pull"
Jul 29 22:58:05.789886 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789889    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="pull"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789898    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="util"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789913    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="util"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789937    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="util"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789942    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="util"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789948    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789953    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="util"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.789965    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="util"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.790015    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cc6a4b-1401-4fbe-a7bd-238cc99f600c" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.790021    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86dd113-ab93-4569-ab79-88b1535325eb" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.790027    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd0956d-1ec7-481b-949e-094805ab5417" containerName="extract"
Jul 29 22:58:05.790299 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.790032    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5892656f-322d-4336-bfe9-f4a94d3aaf6d" containerName="extract"
Jul 29 22:58:05.794431 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.794412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:05.796864 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.796842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mrwzs\""
Jul 29 22:58:05.797012 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.796986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 22:58:05.797562 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.797540    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 22:58:05.799080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.799055    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-b9spq"]
Jul 29 22:58:05.922941 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:05.922897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzhx\" (UniqueName: \"kubernetes.io/projected/9bebaadb-3b3d-476d-8fea-8c83f55be22c-kube-api-access-gmzhx\") pod \"authorino-operator-56fb6f6856-b9spq\" (UID: \"9bebaadb-3b3d-476d-8fea-8c83f55be22c\") " pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:06.024022 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:06.023980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzhx\" (UniqueName: \"kubernetes.io/projected/9bebaadb-3b3d-476d-8fea-8c83f55be22c-kube-api-access-gmzhx\") pod \"authorino-operator-56fb6f6856-b9spq\" (UID: \"9bebaadb-3b3d-476d-8fea-8c83f55be22c\") " pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:06.035280 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:06.035247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzhx\" (UniqueName: \"kubernetes.io/projected/9bebaadb-3b3d-476d-8fea-8c83f55be22c-kube-api-access-gmzhx\") pod \"authorino-operator-56fb6f6856-b9spq\" (UID: \"9bebaadb-3b3d-476d-8fea-8c83f55be22c\") " pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:06.104747 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:06.104714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:06.243430 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:06.243403    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-b9spq"]
Jul 29 22:58:06.245076 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:58:06.245050    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bebaadb_3b3d_476d_8fea_8c83f55be22c.slice/crio-0d9204faba9e5b59997b5f4888b1fed5a6e60c3feff01c52098750c35e5cf748 WatchSource:0}: Error finding container 0d9204faba9e5b59997b5f4888b1fed5a6e60c3feff01c52098750c35e5cf748: Status 404 returned error can't find the container with id 0d9204faba9e5b59997b5f4888b1fed5a6e60c3feff01c52098750c35e5cf748
Jul 29 22:58:07.226332 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:07.226292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq" event={"ID":"9bebaadb-3b3d-476d-8fea-8c83f55be22c","Type":"ContainerStarted","Data":"0d9204faba9e5b59997b5f4888b1fed5a6e60c3feff01c52098750c35e5cf748"}
Jul 29 22:58:09.237890 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:09.237857    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq" event={"ID":"9bebaadb-3b3d-476d-8fea-8c83f55be22c","Type":"ContainerStarted","Data":"12cd2f5e91d58e6a8fb59fa5b033a6fe68bae5231511e22c5c0125ca8b49d706"}
Jul 29 22:58:09.238263 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:09.237919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:09.281084 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:09.281032    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq" podStartSLOduration=1.9784438149999999 podStartE2EDuration="4.281015452s" totalImagesPullingTime="2.302571637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:58:09.279153182 +0000 UTC m=+705.187843539" watchObservedRunningTime="2026-07-29 22:58:09.281015452 +0000 UTC m=+705.189705810"
Jul 29 22:58:12.673085 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.673051    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"]
Jul 29 22:58:12.681564 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.681537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:12.683953 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.683930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s4d7l\""
Jul 29 22:58:12.683953 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.683945    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 29 22:58:12.691391 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.691370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"]
Jul 29 22:58:12.780983 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.780949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb27c\" (UniqueName: \"kubernetes.io/projected/be056854-f42a-407f-bc8e-6bc3b64839d9-kube-api-access-qb27c\") pod \"dns-operator-controller-manager-ccbdf686-dccw7\" (UID: \"be056854-f42a-407f-bc8e-6bc3b64839d9\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:12.881403 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.881368    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb27c\" (UniqueName: \"kubernetes.io/projected/be056854-f42a-407f-bc8e-6bc3b64839d9-kube-api-access-qb27c\") pod \"dns-operator-controller-manager-ccbdf686-dccw7\" (UID: \"be056854-f42a-407f-bc8e-6bc3b64839d9\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:12.891930 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.891906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb27c\" (UniqueName: \"kubernetes.io/projected/be056854-f42a-407f-bc8e-6bc3b64839d9-kube-api-access-qb27c\") pod \"dns-operator-controller-manager-ccbdf686-dccw7\" (UID: \"be056854-f42a-407f-bc8e-6bc3b64839d9\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:12.991903 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:12.991821    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:13.130667 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:13.130639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"]
Jul 29 22:58:13.132615 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:58:13.132584    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe056854_f42a_407f_bc8e_6bc3b64839d9.slice/crio-d97671734d4b8461adf0b6bd1051f8362c40635647cd8081ee7918282724787b WatchSource:0}: Error finding container d97671734d4b8461adf0b6bd1051f8362c40635647cd8081ee7918282724787b: Status 404 returned error can't find the container with id d97671734d4b8461adf0b6bd1051f8362c40635647cd8081ee7918282724787b
Jul 29 22:58:13.259616 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:13.259538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7" event={"ID":"be056854-f42a-407f-bc8e-6bc3b64839d9","Type":"ContainerStarted","Data":"d97671734d4b8461adf0b6bd1051f8362c40635647cd8081ee7918282724787b"}
Jul 29 22:58:15.278039 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:15.277938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7" event={"ID":"be056854-f42a-407f-bc8e-6bc3b64839d9","Type":"ContainerStarted","Data":"5cd7a5732a0503602ec70609a7280e94be0e626be11f9d636dfea73341ad614c"}
Jul 29 22:58:15.278498 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:15.278083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:15.296904 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:15.296855    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7" podStartSLOduration=1.538331337 podStartE2EDuration="3.296840549s" totalImagesPullingTime="1.758509212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:58:15.295600674 +0000 UTC m=+711.204291031" watchObservedRunningTime="2026-07-29 22:58:15.296840549 +0000 UTC m=+711.205530906"
Jul 29 22:58:18.911874 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:18.911764    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"]
Jul 29 22:58:18.915677 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:18.915656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:18.918755 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:18.918740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8pc7g\""
Jul 29 22:58:18.927609 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:18.927587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"]
Jul 29 22:58:19.033970 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.033929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70dc8007-2ad5-4595-a170-adb08518b5e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-tpm8g\" (UID: \"70dc8007-2ad5-4595-a170-adb08518b5e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.034162 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.034003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-str75\" (UniqueName: \"kubernetes.io/projected/70dc8007-2ad5-4595-a170-adb08518b5e1-kube-api-access-str75\") pod \"kuadrant-operator-controller-manager-66866f5fcc-tpm8g\" (UID: \"70dc8007-2ad5-4595-a170-adb08518b5e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.135343 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.135300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-str75\" (UniqueName: \"kubernetes.io/projected/70dc8007-2ad5-4595-a170-adb08518b5e1-kube-api-access-str75\") pod \"kuadrant-operator-controller-manager-66866f5fcc-tpm8g\" (UID: \"70dc8007-2ad5-4595-a170-adb08518b5e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.135540 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.135375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70dc8007-2ad5-4595-a170-adb08518b5e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-tpm8g\" (UID: \"70dc8007-2ad5-4595-a170-adb08518b5e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.135765 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.135741    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70dc8007-2ad5-4595-a170-adb08518b5e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-tpm8g\" (UID: \"70dc8007-2ad5-4595-a170-adb08518b5e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.148970 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.148938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-str75\" (UniqueName: \"kubernetes.io/projected/70dc8007-2ad5-4595-a170-adb08518b5e1-kube-api-access-str75\") pod \"kuadrant-operator-controller-manager-66866f5fcc-tpm8g\" (UID: \"70dc8007-2ad5-4595-a170-adb08518b5e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.226011 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.225924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:19.360649 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:19.360628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"]
Jul 29 22:58:19.362511 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:58:19.362479    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dc8007_2ad5_4595_a170_adb08518b5e1.slice/crio-40c92560328ac837c910265af67810eb918c551dbe256d6fb1a1101d255bede5 WatchSource:0}: Error finding container 40c92560328ac837c910265af67810eb918c551dbe256d6fb1a1101d255bede5: Status 404 returned error can't find the container with id 40c92560328ac837c910265af67810eb918c551dbe256d6fb1a1101d255bede5
Jul 29 22:58:20.246163 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:20.246131    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-b9spq"
Jul 29 22:58:20.309550 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:20.309513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g" event={"ID":"70dc8007-2ad5-4595-a170-adb08518b5e1","Type":"ContainerStarted","Data":"40c92560328ac837c910265af67810eb918c551dbe256d6fb1a1101d255bede5"}
Jul 29 22:58:24.334080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:24.334043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g" event={"ID":"70dc8007-2ad5-4595-a170-adb08518b5e1","Type":"ContainerStarted","Data":"eb64bd38ef60e6dc78e037d47ee4c8dd8a017a840d1979661f56222246f8dc42"}
Jul 29 22:58:24.334502 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:24.334153    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:58:24.356464 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:24.356414    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g" podStartSLOduration=2.439789546 podStartE2EDuration="6.356393365s" totalImagesPullingTime="3.916603819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:58:24.354906297 +0000 UTC m=+720.263596655" watchObservedRunningTime="2026-07-29 22:58:24.356393365 +0000 UTC m=+720.265083725"
Jul 29 22:58:26.289520 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:26.289491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-dccw7"
Jul 29 22:58:35.340382 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:58:35.340347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-tpm8g"
Jul 29 22:59:10.712755 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.712718    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"]
Jul 29 22:59:10.717304 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.717285    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:10.719524 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.719503    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 22:59:10.719642 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.719563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m864v\""
Jul 29 22:59:10.725306 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.725284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"]
Jul 29 22:59:10.759597 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.759561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"]
Jul 29 22:59:10.778907 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.778866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebce846c-4a18-47d1-aa6f-5aca4a7756ba-config-file\") pod \"limitador-limitador-85d9db4d5d-rn5v2\" (UID: \"ebce846c-4a18-47d1-aa6f-5aca4a7756ba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:10.779074 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.778995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47pg\" (UniqueName: \"kubernetes.io/projected/ebce846c-4a18-47d1-aa6f-5aca4a7756ba-kube-api-access-w47pg\") pod \"limitador-limitador-85d9db4d5d-rn5v2\" (UID: \"ebce846c-4a18-47d1-aa6f-5aca4a7756ba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:10.879625 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.879595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47pg\" (UniqueName: \"kubernetes.io/projected/ebce846c-4a18-47d1-aa6f-5aca4a7756ba-kube-api-access-w47pg\") pod \"limitador-limitador-85d9db4d5d-rn5v2\" (UID: \"ebce846c-4a18-47d1-aa6f-5aca4a7756ba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:10.879810 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.879658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebce846c-4a18-47d1-aa6f-5aca4a7756ba-config-file\") pod \"limitador-limitador-85d9db4d5d-rn5v2\" (UID: \"ebce846c-4a18-47d1-aa6f-5aca4a7756ba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:10.880288 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.880269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebce846c-4a18-47d1-aa6f-5aca4a7756ba-config-file\") pod \"limitador-limitador-85d9db4d5d-rn5v2\" (UID: \"ebce846c-4a18-47d1-aa6f-5aca4a7756ba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:10.887398 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:10.887370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47pg\" (UniqueName: \"kubernetes.io/projected/ebce846c-4a18-47d1-aa6f-5aca4a7756ba-kube-api-access-w47pg\") pod \"limitador-limitador-85d9db4d5d-rn5v2\" (UID: \"ebce846c-4a18-47d1-aa6f-5aca4a7756ba\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:11.028120 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:11.028033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:11.174142 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:11.174115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"]
Jul 29 22:59:11.175879 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:59:11.175841    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebce846c_4a18_47d1_aa6f_5aca4a7756ba.slice/crio-5440c4cc1ce45a166fdb4d9b69de3e96aa3c6132b292619fa104b61d285ad96e WatchSource:0}: Error finding container 5440c4cc1ce45a166fdb4d9b69de3e96aa3c6132b292619fa104b61d285ad96e: Status 404 returned error can't find the container with id 5440c4cc1ce45a166fdb4d9b69de3e96aa3c6132b292619fa104b61d285ad96e
Jul 29 22:59:11.575084 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:11.575052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2" event={"ID":"ebce846c-4a18-47d1-aa6f-5aca4a7756ba","Type":"ContainerStarted","Data":"5440c4cc1ce45a166fdb4d9b69de3e96aa3c6132b292619fa104b61d285ad96e"}
Jul 29 22:59:12.581360 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:12.581325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2" event={"ID":"ebce846c-4a18-47d1-aa6f-5aca4a7756ba","Type":"ContainerStarted","Data":"64a1576b513b570d56ca16b7a65aa10a9e6fc5be2ea7cfc9289db8431900a06a"}
Jul 29 22:59:12.581848 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:12.581434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:12.599501 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:12.599454    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2" podStartSLOduration=1.327162714 podStartE2EDuration="2.599439453s" totalImagesPullingTime="1.272276739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:12.596827148 +0000 UTC m=+768.505517506" watchObservedRunningTime="2026-07-29 22:59:12.599439453 +0000 UTC m=+768.508129811"
Jul 29 22:59:23.587152 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:23.587123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-rn5v2"
Jul 29 22:59:45.512055 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.512018    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-qq7rx"]
Jul 29 22:59:45.512555 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.512226    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-844cf74686-qq7rx" podUID="309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" containerName="manager" containerID="cri-o://269680a2dac791a772c37ad67fd47903eced8cc33d16b8ed471067ca15df6367" gracePeriod=10
Jul 29 22:59:45.536860 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.536836    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-844cf74686-w286v"]
Jul 29 22:59:45.540139 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.540122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.548741 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.548711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-w286v"]
Jul 29 22:59:45.701378 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.701349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jd6p\" (UniqueName: \"kubernetes.io/projected/ced5244f-c645-46e2-ab45-38b672cab11b-kube-api-access-6jd6p\") pod \"kserve-controller-manager-844cf74686-w286v\" (UID: \"ced5244f-c645-46e2-ab45-38b672cab11b\") " pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.701518 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.701416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ced5244f-c645-46e2-ab45-38b672cab11b-cert\") pod \"kserve-controller-manager-844cf74686-w286v\" (UID: \"ced5244f-c645-46e2-ab45-38b672cab11b\") " pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.753345 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.753302    2580 generic.go:356] "Generic (PLEG): container finished" podID="309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" containerID="269680a2dac791a772c37ad67fd47903eced8cc33d16b8ed471067ca15df6367" exitCode=0
Jul 29 22:59:45.753533 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.753366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844cf74686-qq7rx" event={"ID":"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3","Type":"ContainerDied","Data":"269680a2dac791a772c37ad67fd47903eced8cc33d16b8ed471067ca15df6367"}
Jul 29 22:59:45.767337 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.767281    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:59:45.802918 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.802882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jd6p\" (UniqueName: \"kubernetes.io/projected/ced5244f-c645-46e2-ab45-38b672cab11b-kube-api-access-6jd6p\") pod \"kserve-controller-manager-844cf74686-w286v\" (UID: \"ced5244f-c645-46e2-ab45-38b672cab11b\") " pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.803095 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.802975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ced5244f-c645-46e2-ab45-38b672cab11b-cert\") pod \"kserve-controller-manager-844cf74686-w286v\" (UID: \"ced5244f-c645-46e2-ab45-38b672cab11b\") " pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.805523 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.805495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ced5244f-c645-46e2-ab45-38b672cab11b-cert\") pod \"kserve-controller-manager-844cf74686-w286v\" (UID: \"ced5244f-c645-46e2-ab45-38b672cab11b\") " pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.811343 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.811316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jd6p\" (UniqueName: \"kubernetes.io/projected/ced5244f-c645-46e2-ab45-38b672cab11b-kube-api-access-6jd6p\") pod \"kserve-controller-manager-844cf74686-w286v\" (UID: \"ced5244f-c645-46e2-ab45-38b672cab11b\") " pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.903878 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.903851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-cert\") pod \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") "
Jul 29 22:59:45.904042 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.903956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2hd4\" (UniqueName: \"kubernetes.io/projected/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-kube-api-access-b2hd4\") pod \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\" (UID: \"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3\") "
Jul 29 22:59:45.904624 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.904605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:45.906144 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.906114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-cert" (OuterVolumeSpecName: "cert") pod "309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" (UID: "309355c0-fbc5-42b4-bdaf-9bf04a50e2f3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:59:45.906237 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:45.906144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-kube-api-access-b2hd4" (OuterVolumeSpecName: "kube-api-access-b2hd4") pod "309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" (UID: "309355c0-fbc5-42b4-bdaf-9bf04a50e2f3"). InnerVolumeSpecName "kube-api-access-b2hd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:59:46.004739 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.004698    2580 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:59:46.004739 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.004733    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2hd4\" (UniqueName: \"kubernetes.io/projected/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3-kube-api-access-b2hd4\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 22:59:46.036544 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.036521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-w286v"]
Jul 29 22:59:46.038247 ip-10-0-131-193 kubenswrapper[2580]: W0729 22:59:46.038223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced5244f_c645_46e2_ab45_38b672cab11b.slice/crio-713579cc8693ebb273862f9cea9063c0e2d5d88c7e6cd5fe80bf2730a8a7d24b WatchSource:0}: Error finding container 713579cc8693ebb273862f9cea9063c0e2d5d88c7e6cd5fe80bf2730a8a7d24b: Status 404 returned error can't find the container with id 713579cc8693ebb273862f9cea9063c0e2d5d88c7e6cd5fe80bf2730a8a7d24b
Jul 29 22:59:46.761114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.761034    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-844cf74686-qq7rx"
Jul 29 22:59:46.761114 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.761057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844cf74686-qq7rx" event={"ID":"309355c0-fbc5-42b4-bdaf-9bf04a50e2f3","Type":"ContainerDied","Data":"72ebdf4f33c2e428e30e545cc824571902e4d8661cd9397faa486360322b4891"}
Jul 29 22:59:46.761597 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.761134    2580 scope.go:117] "RemoveContainer" containerID="269680a2dac791a772c37ad67fd47903eced8cc33d16b8ed471067ca15df6367"
Jul 29 22:59:46.762667 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.762644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844cf74686-w286v" event={"ID":"ced5244f-c645-46e2-ab45-38b672cab11b","Type":"ContainerStarted","Data":"66a748a33ffcd0ad034b17ac3eee84618a722bfb8865960d4c51a96ceb399d86"}
Jul 29 22:59:46.762771 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.762691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-844cf74686-w286v" event={"ID":"ced5244f-c645-46e2-ab45-38b672cab11b","Type":"ContainerStarted","Data":"713579cc8693ebb273862f9cea9063c0e2d5d88c7e6cd5fe80bf2730a8a7d24b"}
Jul 29 22:59:46.762872 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.762858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 22:59:46.776582 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.776560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-qq7rx"]
Jul 29 22:59:46.779870 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.779850    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-844cf74686-qq7rx"]
Jul 29 22:59:46.792080 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:46.792034    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-844cf74686-w286v" podStartSLOduration=1.353948959 podStartE2EDuration="1.792020797s" totalImagesPullingTime="438.071838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:59:46.790389683 +0000 UTC m=+802.699080042" watchObservedRunningTime="2026-07-29 22:59:46.792020797 +0000 UTC m=+802.700711202"
Jul 29 22:59:48.569542 ip-10-0-131-193 kubenswrapper[2580]: I0729 22:59:48.569509    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" path="/var/lib/kubelet/pods/309355c0-fbc5-42b4-bdaf-9bf04a50e2f3/volumes"
Jul 29 23:00:17.773848 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:00:17.773818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-844cf74686-w286v"
Jul 29 23:02:27.002500 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.002467    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"]
Jul 29 23:02:27.003090 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.003063    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" containerName="manager"
Jul 29 23:02:27.003090 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.003091    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" containerName="manager"
Jul 29 23:02:27.003257 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.003196    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="309355c0-fbc5-42b4-bdaf-9bf04a50e2f3" containerName="manager"
Jul 29 23:02:27.005820 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.005803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.008244 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.008212    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 29 23:02:27.008370 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.008263    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-92l78\""
Jul 29 23:02:27.008370 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.008311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 29 23:02:27.008370 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.008320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 29 23:02:27.008370 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.008216    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 29 23:02:27.016511 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.016482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"]
Jul 29 23:02:27.137819 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.137767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.138025 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.137830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.138025 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.137893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca921f-332d-4d31-9475-e9600c2f5765-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.138025 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.137935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.138025 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.137994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.138227 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.138091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.138227 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.138127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879n9\" (UniqueName: \"kubernetes.io/projected/31ca921f-332d-4d31-9475-e9600c2f5765-kube-api-access-879n9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.239546 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879n9\" (UniqueName: \"kubernetes.io/projected/31ca921f-332d-4d31-9475-e9600c2f5765-kube-api-access-879n9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.239546 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.239821 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.239821 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca921f-332d-4d31-9475-e9600c2f5765-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.239942 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.239942 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.240043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.239994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.240285 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.240259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.240407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.240293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.240407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.240353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.240407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.240373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.242077 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.242057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.242294 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.242268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca921f-332d-4d31-9475-e9600c2f5765-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.247542 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.247517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-879n9\" (UniqueName: \"kubernetes.io/projected/31ca921f-332d-4d31-9475-e9600c2f5765-kube-api-access-879n9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.319381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.319339    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:02:27.494373 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.494269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"]
Jul 29 23:02:27.504731 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:02:27.504702    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca921f_332d_4d31_9475_e9600c2f5765.slice/crio-deb5f97b7acf25ce400273d6ec8baf3a18bbd386b1d3eefc1f8b91d4e3091e67 WatchSource:0}: Error finding container deb5f97b7acf25ce400273d6ec8baf3a18bbd386b1d3eefc1f8b91d4e3091e67: Status 404 returned error can't find the container with id deb5f97b7acf25ce400273d6ec8baf3a18bbd386b1d3eefc1f8b91d4e3091e67
Jul 29 23:02:27.506978 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.506962    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:02:27.620820 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:27.620720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" event={"ID":"31ca921f-332d-4d31-9475-e9600c2f5765","Type":"ContainerStarted","Data":"deb5f97b7acf25ce400273d6ec8baf3a18bbd386b1d3eefc1f8b91d4e3091e67"}
Jul 29 23:02:31.653483 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:31.653394    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" event={"ID":"31ca921f-332d-4d31-9475-e9600c2f5765","Type":"ContainerStarted","Data":"7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06"}
Jul 29 23:02:32.096820 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.096761    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 29 23:02:32.097005 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.096850    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podUID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 29 23:02:32.664856 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.664812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" event={"ID":"6e724f3c-88f2-4fcc-924e-95054f341fd2","Type":"ContainerDied","Data":"659fde2e5fe70ee279988edd6e2798fd625d2bf00d7d50bfbef3e81040a5100a"}
Jul 29 23:02:32.665306 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.664829    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerID="659fde2e5fe70ee279988edd6e2798fd625d2bf00d7d50bfbef3e81040a5100a" exitCode=0
Jul 29 23:02:32.665306 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.665297    2580 scope.go:117] "RemoveContainer" containerID="659fde2e5fe70ee279988edd6e2798fd625d2bf00d7d50bfbef3e81040a5100a"
Jul 29 23:02:32.666824 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.666781    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 23:02:32.666890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.666876    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 23:02:32.666934 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:32.666921    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 29 23:02:33.672602 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:33.672557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" event={"ID":"6e724f3c-88f2-4fcc-924e-95054f341fd2","Type":"ContainerStarted","Data":"5f4a5b905cb859b5285fa583a79f34be2d9a7e9c64d13aef4367f0558f6865e2"}
Jul 29 23:02:34.346259 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:34.346212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 23:02:34.346529 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:34.346500    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 29 23:02:34.346628 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:34.346561    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podUID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 29 23:02:35.346282 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:35.346243    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 29 23:02:35.346677 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:35.346330    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podUID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 29 23:02:36.346031 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:36.346001    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 29 23:02:36.346271 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:36.346250    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podUID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 29 23:02:37.345998 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:37.345960    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 29 23:02:37.346392 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:37.346017    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podUID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 29 23:02:37.704157 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:37.704071    2580 generic.go:356] "Generic (PLEG): container finished" podID="31ca921f-332d-4d31-9475-e9600c2f5765" containerID="7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06" exitCode=0
Jul 29 23:02:37.704157 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:37.704139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" event={"ID":"31ca921f-332d-4d31-9475-e9600c2f5765","Type":"ContainerDied","Data":"7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06"}
Jul 29 23:02:38.346067 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:38.346028    2580 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 29 23:02:38.346527 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:38.346092    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw" podUID="6e724f3c-88f2-4fcc-924e-95054f341fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 29 23:02:39.352407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:39.351763    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 23:02:39.352407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:39.352251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 23:02:39.353704 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:02:39.353643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw"
Jul 29 23:03:07.913288 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:07.913256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" event={"ID":"31ca921f-332d-4d31-9475-e9600c2f5765","Type":"ContainerStarted","Data":"c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714"}
Jul 29 23:03:07.935480 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:07.935426    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podStartSLOduration=2.709999985 podStartE2EDuration="41.935407052s" totalImagesPullingTime="33.232788504s" totalInitContainerRuntime="5.992618563s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:02:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:03:07.932222162 +0000 UTC m=+1003.840912519" watchObservedRunningTime="2026-07-29 23:03:07.935407052 +0000 UTC m=+1003.844097410"
Jul 29 23:03:08.320449 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:08.320413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:03:17.319601 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:17.319561    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:03:17.321178 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:17.321151    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:03:27.320071 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:27.320033    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:03:37.319764 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:37.319725    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:03:47.319964 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:47.319919    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:03:57.319913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:03:57.319872    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:04:07.320859 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:04:07.320806    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:04:17.320701 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:04:17.320599    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:04:27.320485 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:04:27.320446    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:04:37.320678 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:04:37.320636    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:04:47.320236 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:04:47.320186    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:04:57.320684 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:04:57.320643    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 29 23:05:07.330157 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:07.330123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:05:07.338099 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:07.338074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:05:30.616610 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:30.616575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"]
Jul 29 23:05:45.639140 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:45.639094    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" containerID="cri-o://c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714" gracePeriod=45
Jul 29 23:05:46.412574 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.412522    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:05:46.570803 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-879n9\" (UniqueName: \"kubernetes.io/projected/31ca921f-332d-4d31-9475-e9600c2f5765-kube-api-access-879n9\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.570988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-kserve-provision-location\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.570988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-tmp-dir\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.570988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-model-cache\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.570988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-dshm\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.570988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570936    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca921f-332d-4d31-9475-e9600c2f5765-tls-certs\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.570988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.570986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-home\") pod \"31ca921f-332d-4d31-9475-e9600c2f5765\" (UID: \"31ca921f-332d-4d31-9475-e9600c2f5765\") "
Jul 29 23:05:46.571316 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.571159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:46.571316 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.571251    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.571780 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.571754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-home" (OuterVolumeSpecName: "home") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:46.573089 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.573066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-dshm" (OuterVolumeSpecName: "dshm") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:46.573189 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.573174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ca921f-332d-4d31-9475-e9600c2f5765-kube-api-access-879n9" (OuterVolumeSpecName: "kube-api-access-879n9") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "kube-api-access-879n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:05:46.573269 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.573246    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ca921f-332d-4d31-9475-e9600c2f5765-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:05:46.589273 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.589242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:46.633405 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.633343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31ca921f-332d-4d31-9475-e9600c2f5765" (UID: "31ca921f-332d-4d31-9475-e9600c2f5765"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:05:46.672681 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.672589    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879n9\" (UniqueName: \"kubernetes.io/projected/31ca921f-332d-4d31-9475-e9600c2f5765-kube-api-access-879n9\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.672681 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.672627    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.672681 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.672641    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.672681 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.672653    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.672681 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.672666    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca921f-332d-4d31-9475-e9600c2f5765-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.672681 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.672676    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca921f-332d-4d31-9475-e9600c2f5765-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:05:46.794901 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.794868    2580 generic.go:356] "Generic (PLEG): container finished" podID="31ca921f-332d-4d31-9475-e9600c2f5765" containerID="c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714" exitCode=0
Jul 29 23:05:46.795080 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.794924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"
Jul 29 23:05:46.795080 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.794932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" event={"ID":"31ca921f-332d-4d31-9475-e9600c2f5765","Type":"ContainerDied","Data":"c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714"}
Jul 29 23:05:46.795080 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.794975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" event={"ID":"31ca921f-332d-4d31-9475-e9600c2f5765","Type":"ContainerDied","Data":"deb5f97b7acf25ce400273d6ec8baf3a18bbd386b1d3eefc1f8b91d4e3091e67"}
Jul 29 23:05:46.795080 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.795000    2580 scope.go:117] "RemoveContainer" containerID="c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714"
Jul 29 23:05:46.804727 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.804708    2580 scope.go:117] "RemoveContainer" containerID="7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06"
Jul 29 23:05:46.815341 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.815315    2580 scope.go:117] "RemoveContainer" containerID="c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714"
Jul 29 23:05:46.816060 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:05:46.815810    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714\": container with ID starting with c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714 not found: ID does not exist" containerID="c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714"
Jul 29 23:05:46.816060 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.815850    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714"} err="failed to get container status \"c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714\": rpc error: code = NotFound desc = could not find container \"c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714\": container with ID starting with c836a4490946f318785b19a7f9c217f5486aa2e7e62aa6fd8867f56028485714 not found: ID does not exist"
Jul 29 23:05:46.816060 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.815875    2580 scope.go:117] "RemoveContainer" containerID="7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06"
Jul 29 23:05:46.816261 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:05:46.816141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06\": container with ID starting with 7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06 not found: ID does not exist" containerID="7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06"
Jul 29 23:05:46.816261 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.816181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06"} err="failed to get container status \"7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06\": rpc error: code = NotFound desc = could not find container \"7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06\": container with ID starting with 7a57ad9aac3f7ebf6aa976f0ca88e4f1651ae5bc53f320ed5f28a3037c771c06 not found: ID does not exist"
Jul 29 23:05:46.818114 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.818094    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"]
Jul 29 23:05:46.821230 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:46.821210    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn"]
Jul 29 23:05:47.331449 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:47.331401    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5847fb6b7cmznvn" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 29 23:05:48.569772 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:05:48.569739    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" path="/var/lib/kubelet/pods/31ca921f-332d-4d31-9475-e9600c2f5765/volumes"
Jul 29 23:06:03.082373 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.082339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 23:06:03.082763 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.082700    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="storage-initializer"
Jul 29 23:06:03.082763 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.082710    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="storage-initializer"
Jul 29 23:06:03.082763 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.082723    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main"
Jul 29 23:06:03.082763 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.082729    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main"
Jul 29 23:06:03.083006 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.082818    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ca921f-332d-4d31-9475-e9600c2f5765" containerName="main"
Jul 29 23:06:03.085835 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.085817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.089396 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.089370    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-h8vhb\""
Jul 29 23:06:03.089701 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.089679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 29 23:06:03.089883 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.089687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 29 23:06:03.090173 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.090143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-lhrnv\""
Jul 29 23:06:03.090173 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.090158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 29 23:06:03.090441 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.090418    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 29 23:06:03.103828 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.103803    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 23:06:03.216038 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.216242 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.216242 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c519f0-d3ae-4e13-af88-82f626b70542-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.216242 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.216441 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.216441 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsppt\" (UniqueName: \"kubernetes.io/projected/e6c519f0-d3ae-4e13-af88-82f626b70542-kube-api-access-xsppt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.216441 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.216328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.317578 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.317756 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c519f0-d3ae-4e13-af88-82f626b70542-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.317756 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.317756 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.317917 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsppt\" (UniqueName: \"kubernetes.io/projected/e6c519f0-d3ae-4e13-af88-82f626b70542-kube-api-access-xsppt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.317917 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.318020 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.318020 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.317999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.318108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.318041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.318154 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.318108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.318450 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.318432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.320010 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.319986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.320096 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.320031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c519f0-d3ae-4e13-af88-82f626b70542-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.325999 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.325979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsppt\" (UniqueName: \"kubernetes.io/projected/e6c519f0-d3ae-4e13-af88-82f626b70542-kube-api-access-xsppt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.398744 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.398663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:03.544066 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.544036    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 23:06:03.546085 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:06:03.546055    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c519f0_d3ae_4e13_af88_82f626b70542.slice/crio-ff4ade03a961082837a87259e4015868cec9f7f9f11cdc4dc54527cab19e6b9f WatchSource:0}: Error finding container ff4ade03a961082837a87259e4015868cec9f7f9f11cdc4dc54527cab19e6b9f: Status 404 returned error can't find the container with id ff4ade03a961082837a87259e4015868cec9f7f9f11cdc4dc54527cab19e6b9f
Jul 29 23:06:03.890102 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.890061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e6c519f0-d3ae-4e13-af88-82f626b70542","Type":"ContainerStarted","Data":"021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d"}
Jul 29 23:06:03.890320 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:03.890119    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e6c519f0-d3ae-4e13-af88-82f626b70542","Type":"ContainerStarted","Data":"ff4ade03a961082837a87259e4015868cec9f7f9f11cdc4dc54527cab19e6b9f"}
Jul 29 23:06:08.923478 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:08.923437    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerID="021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d" exitCode=0
Jul 29 23:06:08.924026 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:08.923513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e6c519f0-d3ae-4e13-af88-82f626b70542","Type":"ContainerDied","Data":"021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d"}
Jul 29 23:06:09.932641 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:09.932609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e6c519f0-d3ae-4e13-af88-82f626b70542","Type":"ContainerStarted","Data":"a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679"}
Jul 29 23:06:09.952381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:09.952323    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.271365515 podStartE2EDuration="6.952304292s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.680938777s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:06:09.951438289 +0000 UTC m=+1185.860128671" watchObservedRunningTime="2026-07-29 23:06:09.952304292 +0000 UTC m=+1185.860994652"
Jul 29 23:06:10.399052 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:10.399019    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:13.399125 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:13.399084    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:06:13.400854 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:13.400820    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:06:23.400126 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:23.400082    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:06:33.399280 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:33.399229    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:06:43.399434 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:43.399382    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:06:53.400024 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:06:53.399977    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:07:03.399972 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:07:03.399928    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:07:13.399135 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:07:13.399092    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:07:23.400267 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:07:23.400156    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:07:33.399533 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:07:33.399484    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:07:43.409895 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:07:43.409861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:07:43.417849 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:07:43.417821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:08:38.963231 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:38.963198    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 23:08:53.988246 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:53.988163    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" containerID="cri-o://a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679" gracePeriod=45
Jul 29 23:08:54.411058 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.411019    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 29 23:08:54.501888 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:08:54.501857    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c519f0_d3ae_4e13_af88_82f626b70542.slice/crio-a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 23:08:54.670539 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.670480    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:08:54.783390 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-model-cache\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783583 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-home\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783583 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-tmp-dir\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783583 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783476    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsppt\" (UniqueName: \"kubernetes.io/projected/e6c519f0-d3ae-4e13-af88-82f626b70542-kube-api-access-xsppt\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783583 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c519f0-d3ae-4e13-af88-82f626b70542-tls-certs\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783583 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-kserve-provision-location\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783583 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-dshm\") pod \"e6c519f0-d3ae-4e13-af88-82f626b70542\" (UID: \"e6c519f0-d3ae-4e13-af88-82f626b70542\") "
Jul 29 23:08:54.783948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:54.783948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.783869    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.784112 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.784085    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-home" (OuterVolumeSpecName: "home") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:54.785904 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.785860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-dshm" (OuterVolumeSpecName: "dshm") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:54.786163 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.786141    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c519f0-d3ae-4e13-af88-82f626b70542-kube-api-access-xsppt" (OuterVolumeSpecName: "kube-api-access-xsppt") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "kube-api-access-xsppt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:08:54.786241 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.786165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c519f0-d3ae-4e13-af88-82f626b70542-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:08:54.801182 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.801144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:54.849354 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.849320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6c519f0-d3ae-4e13-af88-82f626b70542" (UID: "e6c519f0-d3ae-4e13-af88-82f626b70542"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:08:54.851696 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.851671    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerID="a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679" exitCode=0
Jul 29 23:08:54.851849 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.851732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 23:08:54.851849 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.851748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e6c519f0-d3ae-4e13-af88-82f626b70542","Type":"ContainerDied","Data":"a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679"}
Jul 29 23:08:54.851849 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.851819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e6c519f0-d3ae-4e13-af88-82f626b70542","Type":"ContainerDied","Data":"ff4ade03a961082837a87259e4015868cec9f7f9f11cdc4dc54527cab19e6b9f"}
Jul 29 23:08:54.851849 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.851846    2580 scope.go:117] "RemoveContainer" containerID="a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679"
Jul 29 23:08:54.861324 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.861307    2580 scope.go:117] "RemoveContainer" containerID="021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d"
Jul 29 23:08:54.869474 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.869459    2580 scope.go:117] "RemoveContainer" containerID="a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679"
Jul 29 23:08:54.869719 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:08:54.869688    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679\": container with ID starting with a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679 not found: ID does not exist" containerID="a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679"
Jul 29 23:08:54.869818 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.869720    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679"} err="failed to get container status \"a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679\": rpc error: code = NotFound desc = could not find container \"a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679\": container with ID starting with a88a6643c072fb41a816701262040984434e07f5ec1f66eb40d6fac320145679 not found: ID does not exist"
Jul 29 23:08:54.869818 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.869737    2580 scope.go:117] "RemoveContainer" containerID="021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d"
Jul 29 23:08:54.870005 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:08:54.869987    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d\": container with ID starting with 021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d not found: ID does not exist" containerID="021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d"
Jul 29 23:08:54.870047 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.870012    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d"} err="failed to get container status \"021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d\": rpc error: code = NotFound desc = could not find container \"021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d\": container with ID starting with 021b896e69f18ec8d3f1510d53490b0aa4a08287aca9a88813c84f8f6e19890d not found: ID does not exist"
Jul 29 23:08:54.882094 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.882066    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 23:08:54.884647 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.884616    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.884759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.884652    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.884759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.884668    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.884759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.884681    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c519f0-d3ae-4e13-af88-82f626b70542-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.884759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.884702    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsppt\" (UniqueName: \"kubernetes.io/projected/e6c519f0-d3ae-4e13-af88-82f626b70542-kube-api-access-xsppt\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.884759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.884715    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c519f0-d3ae-4e13-af88-82f626b70542-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:08:54.887439 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:54.887411    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 23:08:56.570171 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:08:56.570139    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" path="/var/lib/kubelet/pods/e6c519f0-d3ae-4e13-af88-82f626b70542/volumes"
Jul 29 23:09:05.688279 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.688235    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"]
Jul 29 23:09:05.688853 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.688831    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="storage-initializer"
Jul 29 23:09:05.688931 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.688855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="storage-initializer"
Jul 29 23:09:05.688931 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.688886    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main"
Jul 29 23:09:05.688931 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.688895    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main"
Jul 29 23:09:05.689098 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.688999    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c519f0-d3ae-4e13-af88-82f626b70542" containerName="main"
Jul 29 23:09:05.693049 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.693023    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.695306 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.695282    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 29 23:09:05.695503 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.695285    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 29 23:09:05.695589 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.695334    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 29 23:09:05.695811 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.695441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-vdht2\""
Jul 29 23:09:05.695811 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.695442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 29 23:09:05.703049 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.703025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"]
Jul 29 23:09:05.773381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-model-cache\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.773381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqh5\" (UniqueName: \"kubernetes.io/projected/35657a71-04d3-4882-a5de-17edbf5408ad-kube-api-access-8wqh5\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.773645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-tmp-dir\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.773645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-dshm\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.773645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35657a71-04d3-4882-a5de-17edbf5408ad-tls-certs\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.773645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773526    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-home\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.773645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.773610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874279 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874279 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-model-cache\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874519 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqh5\" (UniqueName: \"kubernetes.io/projected/35657a71-04d3-4882-a5de-17edbf5408ad-kube-api-access-8wqh5\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874519 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-tmp-dir\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874519 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-dshm\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874519 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35657a71-04d3-4882-a5de-17edbf5408ad-tls-certs\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874519 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-home\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874825 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874825 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-model-cache\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874825 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-tmp-dir\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.874967 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.874879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-home\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.876779 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.876760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-dshm\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.877016 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.876994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35657a71-04d3-4882-a5de-17edbf5408ad-tls-certs\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:05.883162 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:05.883140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqh5\" (UniqueName: \"kubernetes.io/projected/35657a71-04d3-4882-a5de-17edbf5408ad-kube-api-access-8wqh5\") pod \"auth-invalid-token-test-kserve-77bfb94c47-wvz9l\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:06.004704 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:06.004619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:06.146160 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:06.146133    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"]
Jul 29 23:09:06.917712 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:06.917669    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" event={"ID":"35657a71-04d3-4882-a5de-17edbf5408ad","Type":"ContainerStarted","Data":"0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3"}
Jul 29 23:09:06.918188 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:06.917733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" event={"ID":"35657a71-04d3-4882-a5de-17edbf5408ad","Type":"ContainerStarted","Data":"bbb80bda6a2c90ac0450cfbb821c9d120f2e8001c88c5cbb5592b0480acd9ab7"}
Jul 29 23:09:12.952181 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:12.952151    2580 generic.go:356] "Generic (PLEG): container finished" podID="35657a71-04d3-4882-a5de-17edbf5408ad" containerID="0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3" exitCode=0
Jul 29 23:09:12.952604 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:12.952225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" event={"ID":"35657a71-04d3-4882-a5de-17edbf5408ad","Type":"ContainerDied","Data":"0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3"}
Jul 29 23:09:13.960845 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:13.960804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" event={"ID":"35657a71-04d3-4882-a5de-17edbf5408ad","Type":"ContainerStarted","Data":"6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f"}
Jul 29 23:09:13.988952 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:13.988893    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podStartSLOduration=2.7252841979999998 podStartE2EDuration="8.98887448s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.263590282s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:09:13.985770837 +0000 UTC m=+1369.894461197" watchObservedRunningTime="2026-07-29 23:09:13.98887448 +0000 UTC m=+1369.897564837"
Jul 29 23:09:14.005647 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:14.005596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:16.005627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:16.005586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:09:16.007494 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:16.007458    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:09:26.005434 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:26.005387    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:09:36.005094 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:36.005047    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:09:46.005487 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:46.005434    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:09:56.005168 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:09:56.005117    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:10:06.005907 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:10:06.005856    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:10:16.005227 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:10:16.005171    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:10:26.005387 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:10:26.005336    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:10:36.005338 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:10:36.005299    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:10:46.005758 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:10:46.005696    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:10:56.005596 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:10:56.005546    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:11:06.015870 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:06.015839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:11:06.024059 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:06.024039    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:11:07.599300 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:07.599268    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"]
Jul 29 23:11:22.665567 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:22.665506    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" containerID="cri-o://6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f" gracePeriod=45
Jul 29 23:11:23.016665 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.016562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 29 23:11:23.410071 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.410046    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:11:23.519411 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-dshm\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.519582 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519435    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-tmp-dir\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.519582 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35657a71-04d3-4882-a5de-17edbf5408ad-tls-certs\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.519653 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519582    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-home\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.519653 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-kserve-provision-location\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.520156 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-model-cache\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.520156 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wqh5\" (UniqueName: \"kubernetes.io/projected/35657a71-04d3-4882-a5de-17edbf5408ad-kube-api-access-8wqh5\") pod \"35657a71-04d3-4882-a5de-17edbf5408ad\" (UID: \"35657a71-04d3-4882-a5de-17edbf5408ad\") "
Jul 29 23:11:23.520156 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.519992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:11:23.520156 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.520119    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.520426 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.520242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-home" (OuterVolumeSpecName: "home") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:11:23.522192 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.522060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35657a71-04d3-4882-a5de-17edbf5408ad-kube-api-access-8wqh5" (OuterVolumeSpecName: "kube-api-access-8wqh5") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "kube-api-access-8wqh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:11:23.522192 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.522089    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-dshm" (OuterVolumeSpecName: "dshm") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:11:23.522192 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.522180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35657a71-04d3-4882-a5de-17edbf5408ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:11:23.537278 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.537201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:11:23.583293 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.583248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35657a71-04d3-4882-a5de-17edbf5408ad" (UID: "35657a71-04d3-4882-a5de-17edbf5408ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:11:23.620972 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.620939    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.620972 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.620967    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.620972 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.620977    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35657a71-04d3-4882-a5de-17edbf5408ad-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.621188 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.620984    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.621188 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.620994    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35657a71-04d3-4882-a5de-17edbf5408ad-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.621188 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.621003    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wqh5\" (UniqueName: \"kubernetes.io/projected/35657a71-04d3-4882-a5de-17edbf5408ad-kube-api-access-8wqh5\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:11:23.727453 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.727422    2580 generic.go:356] "Generic (PLEG): container finished" podID="35657a71-04d3-4882-a5de-17edbf5408ad" containerID="6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f" exitCode=0
Jul 29 23:11:23.727843 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.727485    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"
Jul 29 23:11:23.727843 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.727507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" event={"ID":"35657a71-04d3-4882-a5de-17edbf5408ad","Type":"ContainerDied","Data":"6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f"}
Jul 29 23:11:23.727843 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.727570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l" event={"ID":"35657a71-04d3-4882-a5de-17edbf5408ad","Type":"ContainerDied","Data":"bbb80bda6a2c90ac0450cfbb821c9d120f2e8001c88c5cbb5592b0480acd9ab7"}
Jul 29 23:11:23.727843 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.727593    2580 scope.go:117] "RemoveContainer" containerID="6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f"
Jul 29 23:11:23.739413 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.739391    2580 scope.go:117] "RemoveContainer" containerID="0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3"
Jul 29 23:11:23.747707 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.747687    2580 scope.go:117] "RemoveContainer" containerID="6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f"
Jul 29 23:11:23.747973 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:11:23.747954    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f\": container with ID starting with 6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f not found: ID does not exist" containerID="6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f"
Jul 29 23:11:23.748043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.747981    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f"} err="failed to get container status \"6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f\": rpc error: code = NotFound desc = could not find container \"6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f\": container with ID starting with 6299f7bb214e092744cc90573d120a273108c06587dcb01ead61450a5bd6fb7f not found: ID does not exist"
Jul 29 23:11:23.748043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.747999    2580 scope.go:117] "RemoveContainer" containerID="0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3"
Jul 29 23:11:23.748279 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:11:23.748263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3\": container with ID starting with 0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3 not found: ID does not exist" containerID="0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3"
Jul 29 23:11:23.748328 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.748289    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3"} err="failed to get container status \"0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3\": rpc error: code = NotFound desc = could not find container \"0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3\": container with ID starting with 0d8dfee3a131e5d37f4482edfc7193132e908a151ebab12c3326ea36eaf62cb3 not found: ID does not exist"
Jul 29 23:11:23.751314 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.751294    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"]
Jul 29 23:11:23.755825 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:23.755803    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-77bfb94c47-wvz9l"]
Jul 29 23:11:24.569602 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:24.569569    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" path="/var/lib/kubelet/pods/35657a71-04d3-4882-a5de-17edbf5408ad/volumes"
Jul 29 23:11:43.206569 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.206537    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"]
Jul 29 23:11:43.207131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.206913    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="storage-initializer"
Jul 29 23:11:43.207131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.206924    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="storage-initializer"
Jul 29 23:11:43.207131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.206946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main"
Jul 29 23:11:43.207131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.206953    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main"
Jul 29 23:11:43.207131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.207013    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="35657a71-04d3-4882-a5de-17edbf5408ad" containerName="main"
Jul 29 23:11:43.211529 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.211508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.213694 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.213676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 29 23:11:43.213831 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.213726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-4qvt7\""
Jul 29 23:11:43.213831 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.213679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 29 23:11:43.213992 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.213975    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 29 23:11:43.214073 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.214055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 29 23:11:43.225328 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.225299    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"]
Jul 29 23:11:43.289318 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-home\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.289497 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tncth\" (UniqueName: \"kubernetes.io/projected/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kube-api-access-tncth\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.289497 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kserve-provision-location\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.289497 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-model-cache\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.289614 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tmp-dir\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.289614 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tls-certs\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.289614 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.289541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-dshm\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.390685 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tncth\" (UniqueName: \"kubernetes.io/projected/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kube-api-access-tncth\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.390889 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kserve-provision-location\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.390889 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-model-cache\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.390889 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tmp-dir\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.390889 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tls-certs\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.390889 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-dshm\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.391162 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.390928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-home\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.391239 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.391216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kserve-provision-location\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.391296 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.391262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tmp-dir\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.391296 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.391278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-model-cache\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.391380 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.391308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-home\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.393108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.393089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-dshm\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.393494 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.393474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tls-certs\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.398468 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.398446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tncth\" (UniqueName: \"kubernetes.io/projected/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kube-api-access-tncth\") pod \"auth-disabled-test-kserve-c69bf4cfb-2fvhc\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.523337 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.523248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:43.666461 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.666433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"]
Jul 29 23:11:43.669072 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:11:43.669040    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7710f3_ab86_4eb1_be97_cd9a20616cc8.slice/crio-473090f55217dc6bc50d7f8fa93e5a81707528e3178e42de3dac870011938d40 WatchSource:0}: Error finding container 473090f55217dc6bc50d7f8fa93e5a81707528e3178e42de3dac870011938d40: Status 404 returned error can't find the container with id 473090f55217dc6bc50d7f8fa93e5a81707528e3178e42de3dac870011938d40
Jul 29 23:11:43.834172 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.834131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" event={"ID":"bf7710f3-ab86-4eb1-be97-cd9a20616cc8","Type":"ContainerStarted","Data":"34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23"}
Jul 29 23:11:43.834381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:43.834201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" event={"ID":"bf7710f3-ab86-4eb1-be97-cd9a20616cc8","Type":"ContainerStarted","Data":"473090f55217dc6bc50d7f8fa93e5a81707528e3178e42de3dac870011938d40"}
Jul 29 23:11:48.870262 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:48.870230    2580 generic.go:356] "Generic (PLEG): container finished" podID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerID="34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23" exitCode=0
Jul 29 23:11:48.870679 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:48.870279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" event={"ID":"bf7710f3-ab86-4eb1-be97-cd9a20616cc8","Type":"ContainerDied","Data":"34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23"}
Jul 29 23:11:49.878325 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:49.878282    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" event={"ID":"bf7710f3-ab86-4eb1-be97-cd9a20616cc8","Type":"ContainerStarted","Data":"73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c"}
Jul 29 23:11:49.899534 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:49.899468    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podStartSLOduration=1.842152459 podStartE2EDuration="6.899448559s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.0572961s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:11:49.896568843 +0000 UTC m=+1525.805259201" watchObservedRunningTime="2026-07-29 23:11:49.899448559 +0000 UTC m=+1525.808138918"
Jul 29 23:11:50.524325 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:50.524261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:53.524351 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:53.524309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:11:53.526062 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:11:53.526035    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:12:03.523930 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:03.523883    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:12:13.523955 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:13.523913    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:12:23.523830 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:23.523751    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:12:29.704522 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.704486    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"]
Jul 29 23:12:29.707051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.707028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:29.709222 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.709196    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 29 23:12:29.709727 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.709706    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 29 23:12:29.709868 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.709736    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-xbw9l\""
Jul 29 23:12:29.709868 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.709747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-gqn8x\""
Jul 29 23:12:29.709868 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.709852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 23:12:29.716934 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.716911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"]
Jul 29 23:12:29.809590 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.809560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwc5\" (UniqueName: \"kubernetes.io/projected/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-kube-api-access-fwwc5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:29.809764 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.809599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:29.910107 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.910074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwc5\" (UniqueName: \"kubernetes.io/projected/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-kube-api-access-fwwc5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:29.910271 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.910121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:29.912898 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.912870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:29.917457 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:29.917432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwc5\" (UniqueName: \"kubernetes.io/projected/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-kube-api-access-fwwc5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:30.018562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:30.018468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:30.164665 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:30.164636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"]
Jul 29 23:12:30.166826 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:12:30.166781    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8aadd97_c300_4422_b8d0_3f7cd9b5ba37.slice/crio-89006bf959e34e5aff6bf89bba764cbe879c6f9d809488c29cf7a40ccf0e1521 WatchSource:0}: Error finding container 89006bf959e34e5aff6bf89bba764cbe879c6f9d809488c29cf7a40ccf0e1521: Status 404 returned error can't find the container with id 89006bf959e34e5aff6bf89bba764cbe879c6f9d809488c29cf7a40ccf0e1521
Jul 29 23:12:30.169098 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:30.169075    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:12:31.125458 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:31.125412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" event={"ID":"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37","Type":"ContainerStarted","Data":"89006bf959e34e5aff6bf89bba764cbe879c6f9d809488c29cf7a40ccf0e1521"}
Jul 29 23:12:33.142339 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:33.142293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" event={"ID":"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37","Type":"ContainerStarted","Data":"68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a"}
Jul 29 23:12:33.142813 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:33.142483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:12:33.159370 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:33.159307    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" podStartSLOduration=1.6213334659999998 podStartE2EDuration="4.159285656s" totalImagesPullingTime="2.53795219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:12:33.156538063 +0000 UTC m=+1569.065228416" watchObservedRunningTime="2026-07-29 23:12:33.159285656 +0000 UTC m=+1569.067976014"
Jul 29 23:12:33.524933 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:33.524730    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:12:43.524098 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:43.524055    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:12:53.524326 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:12:53.524284    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:13:03.524340 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:03.524296    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:13:04.150844 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:04.150773    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:13:05.683763 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:05.683727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"]
Jul 29 23:13:05.686304 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:05.686265    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:13:13.523967 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:13.523923    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:13:15.686829 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:15.686771    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 23:13:20.712323 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:20.712231    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main" containerID="cri-o://68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a" gracePeriod=45
Jul 29 23:13:20.979798 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:20.979719    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:13:21.062009 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.061972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwwc5\" (UniqueName: \"kubernetes.io/projected/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-kube-api-access-fwwc5\") pod \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") "
Jul 29 23:13:21.062217 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.062020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-tls-certs\") pod \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\" (UID: \"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37\") "
Jul 29 23:13:21.064246 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.064216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" (UID: "c8aadd97-c300-4422-b8d0-3f7cd9b5ba37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:13:21.064381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.064286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-kube-api-access-fwwc5" (OuterVolumeSpecName: "kube-api-access-fwwc5") pod "c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" (UID: "c8aadd97-c300-4422-b8d0-3f7cd9b5ba37"). InnerVolumeSpecName "kube-api-access-fwwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:13:21.163277 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.163242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwc5\" (UniqueName: \"kubernetes.io/projected/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-kube-api-access-fwwc5\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:13:21.163455 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.163292    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:13:21.414123 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.414088    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerID="68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a" exitCode=0
Jul 29 23:13:21.414311 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.414134    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"
Jul 29 23:13:21.414311 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.414186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" event={"ID":"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37","Type":"ContainerDied","Data":"68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a"}
Jul 29 23:13:21.414311 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.414250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6" event={"ID":"c8aadd97-c300-4422-b8d0-3f7cd9b5ba37","Type":"ContainerDied","Data":"89006bf959e34e5aff6bf89bba764cbe879c6f9d809488c29cf7a40ccf0e1521"}
Jul 29 23:13:21.414311 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.414282    2580 scope.go:117] "RemoveContainer" containerID="68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a"
Jul 29 23:13:21.424267 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.424251    2580 scope.go:117] "RemoveContainer" containerID="68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a"
Jul 29 23:13:21.424532 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:13:21.424515    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a\": container with ID starting with 68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a not found: ID does not exist" containerID="68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a"
Jul 29 23:13:21.424604 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.424541    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a"} err="failed to get container status \"68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a\": rpc error: code = NotFound desc = could not find container \"68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a\": container with ID starting with 68c7aa19aec62bf61c13a5bc24fc372426b28a5e5175968d6fd674dde686f57a not found: ID does not exist"
Jul 29 23:13:21.435494 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.435467    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"]
Jul 29 23:13:21.438704 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:21.438683    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c4867854vv6"]
Jul 29 23:13:22.569652 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:22.569619    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" path="/var/lib/kubelet/pods/c8aadd97-c300-4422-b8d0-3f7cd9b5ba37/volumes"
Jul 29 23:13:23.523972 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:23.523932    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:13:33.524084 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:33.524045    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 29 23:13:43.534619 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:43.534584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:13:43.542744 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:43.542711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:13:59.733754 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:13:59.733717    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"]
Jul 29 23:14:14.755807 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:14.755727    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main" containerID="cri-o://73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c" gracePeriod=45
Jul 29 23:14:15.510356 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.510333    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:14:15.547544 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-model-cache\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547544 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547520    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kserve-provision-location\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547544 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tmp-dir\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547874 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tncth\" (UniqueName: \"kubernetes.io/projected/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kube-api-access-tncth\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547874 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-dshm\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547874 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tls-certs\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547874 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-home\") pod \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\" (UID: \"bf7710f3-ab86-4eb1-be97-cd9a20616cc8\") "
Jul 29 23:14:15.547874 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.547751    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:15.548136 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.548017    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.548427 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.548399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-home" (OuterVolumeSpecName: "home") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:15.550055 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.550025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kube-api-access-tncth" (OuterVolumeSpecName: "kube-api-access-tncth") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "kube-api-access-tncth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:14:15.550573 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.550538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:14:15.550707 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.550684    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-dshm" (OuterVolumeSpecName: "dshm") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:15.562480 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.562449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:15.614655 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.614595    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf7710f3-ab86-4eb1-be97-cd9a20616cc8" (UID: "bf7710f3-ab86-4eb1-be97-cd9a20616cc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:14:15.649025 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.648992    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.649266 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.649240    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.649266 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.649268    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tncth\" (UniqueName: \"kubernetes.io/projected/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-kube-api-access-tncth\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.649401 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.649277    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.649401 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.649287    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.649401 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.649296    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7710f3-ab86-4eb1-be97-cd9a20616cc8-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:14:15.719612 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.719577    2580 generic.go:356] "Generic (PLEG): container finished" podID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerID="73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c" exitCode=0
Jul 29 23:14:15.719612 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.719613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" event={"ID":"bf7710f3-ab86-4eb1-be97-cd9a20616cc8","Type":"ContainerDied","Data":"73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c"}
Jul 29 23:14:15.719903 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.719638    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"
Jul 29 23:14:15.719903 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.719657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc" event={"ID":"bf7710f3-ab86-4eb1-be97-cd9a20616cc8","Type":"ContainerDied","Data":"473090f55217dc6bc50d7f8fa93e5a81707528e3178e42de3dac870011938d40"}
Jul 29 23:14:15.719903 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.719678    2580 scope.go:117] "RemoveContainer" containerID="73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c"
Jul 29 23:14:15.730143 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.730119    2580 scope.go:117] "RemoveContainer" containerID="34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23"
Jul 29 23:14:15.738522 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.738506    2580 scope.go:117] "RemoveContainer" containerID="73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c"
Jul 29 23:14:15.738772 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:14:15.738750    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c\": container with ID starting with 73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c not found: ID does not exist" containerID="73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c"
Jul 29 23:14:15.738841 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.738780    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c"} err="failed to get container status \"73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c\": rpc error: code = NotFound desc = could not find container \"73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c\": container with ID starting with 73eacf4172c1f190350c0e04d4cf1ae2b5db1ea2c42201f8ef4e8ef428e2311c not found: ID does not exist"
Jul 29 23:14:15.738841 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.738814    2580 scope.go:117] "RemoveContainer" containerID="34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23"
Jul 29 23:14:15.739025 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:14:15.739005    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23\": container with ID starting with 34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23 not found: ID does not exist" containerID="34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23"
Jul 29 23:14:15.739083 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.739035    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23"} err="failed to get container status \"34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23\": rpc error: code = NotFound desc = could not find container \"34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23\": container with ID starting with 34d841a6e26656695b8249ee194384e8b5c7da4bba19c79d517c6c1a380edc23 not found: ID does not exist"
Jul 29 23:14:15.744495 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.744471    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"]
Jul 29 23:14:15.750512 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:15.750490    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-c69bf4cfb-2fvhc"]
Jul 29 23:14:16.569369 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:16.569340    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" path="/var/lib/kubelet/pods/bf7710f3-ab86-4eb1-be97-cd9a20616cc8/volumes"
Jul 29 23:14:56.723508 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723473    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"]
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723831    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723857    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="storage-initializer"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723863    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="storage-initializer"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723881    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723887    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723938    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7710f3-ab86-4eb1-be97-cd9a20616cc8" containerName="main"
Jul 29 23:14:56.724023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.723947    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8aadd97-c300-4422-b8d0-3f7cd9b5ba37" containerName="main"
Jul 29 23:14:56.726995 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.726979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.730467 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.730444    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 29 23:14:56.730940 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.730917    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 29 23:14:56.731051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.730929    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-m57ct\""
Jul 29 23:14:56.731051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.730967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 29 23:14:56.731286 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.731268    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-wvckp\""
Jul 29 23:14:56.748497 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.748475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"]
Jul 29 23:14:56.802355 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.802328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.802513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.802364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.802513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.802398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.802513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.802444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9bs\" (UniqueName: \"kubernetes.io/projected/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-kube-api-access-qh9bs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.802513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.802479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.802513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.802505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903446 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9bs\" (UniqueName: \"kubernetes.io/projected/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-kube-api-access-qh9bs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903898 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903965 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.903965 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.904048 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.903969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.905932 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.905906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.906086 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.905998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:56.910919 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:56.910899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9bs\" (UniqueName: \"kubernetes.io/projected/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-kube-api-access-qh9bs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:57.036888 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:57.036804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:14:57.173530 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:57.173497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"]
Jul 29 23:14:57.173945 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:14:57.173916    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76c09e9_3de9_4e99_964d_5a96bbfe54d2.slice/crio-9e3f79d6d89220e334d95fe9d2911b7c29c23cc3ce2b688472097b743ba81ff7 WatchSource:0}: Error finding container 9e3f79d6d89220e334d95fe9d2911b7c29c23cc3ce2b688472097b743ba81ff7: Status 404 returned error can't find the container with id 9e3f79d6d89220e334d95fe9d2911b7c29c23cc3ce2b688472097b743ba81ff7
Jul 29 23:14:57.943616 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:57.943578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" event={"ID":"c76c09e9-3de9-4e99-964d-5a96bbfe54d2","Type":"ContainerStarted","Data":"9e3f79d6d89220e334d95fe9d2911b7c29c23cc3ce2b688472097b743ba81ff7"}
Jul 29 23:14:58.950346 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:58.950310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" event={"ID":"c76c09e9-3de9-4e99-964d-5a96bbfe54d2","Type":"ContainerStarted","Data":"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"}
Jul 29 23:14:58.950859 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:14:58.950405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:15:00.964346 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:00.964307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" event={"ID":"c76c09e9-3de9-4e99-964d-5a96bbfe54d2","Type":"ContainerStarted","Data":"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"}
Jul 29 23:15:00.983892 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:00.983840    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" podStartSLOduration=1.840480031 podStartE2EDuration="4.98382253s" totalImagesPullingTime="3.143342499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:14:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:15:00.981755035 +0000 UTC m=+1716.890445433" watchObservedRunningTime="2026-07-29 23:15:00.98382253 +0000 UTC m=+1716.892512889"
Jul 29 23:15:01.037228 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:01.037197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:15:07.037581 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:07.037542    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:15:07.050273 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:07.050246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:15:08.012170 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:08.012139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:15:08.013346 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:15:08.013315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:16:30.441196 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:30.441160    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"]
Jul 29 23:16:30.451141 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:16:30.451097    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:16:30.451141 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:16:30.451141 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:16:30.451141 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2" cmd=["/bin/sleep","15"]
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:30.451157    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" message="" err=<
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:16:30.451170    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="main" containerID="cri-o://d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"
Jul 29 23:16:30.451395 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:30.451216    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="main" containerID="cri-o://d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2" gracePeriod=60
Jul 29 23:16:30.667619 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:30.667588    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="llm-d-routing-sidecar" containerID="cri-o://37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418" gracePeriod=60
Jul 29 23:16:30.953975 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:30.953950    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:16:31.042560 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-dshm\") pod \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") "
Jul 29 23:16:31.042560 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-home\") pod \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") "
Jul 29 23:16:31.042560 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042541    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-model-cache\") pod \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") "
Jul 29 23:16:31.042886 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tmp-dir\") pod \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") "
Jul 29 23:16:31.042886 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh9bs\" (UniqueName: \"kubernetes.io/projected/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-kube-api-access-qh9bs\") pod \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") "
Jul 29 23:16:31.042886 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tls-certs\") pod \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\" (UID: \"c76c09e9-3de9-4e99-964d-5a96bbfe54d2\") "
Jul 29 23:16:31.042886 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042820    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-home" (OuterVolumeSpecName: "home") pod "c76c09e9-3de9-4e99-964d-5a96bbfe54d2" (UID: "c76c09e9-3de9-4e99-964d-5a96bbfe54d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.042886 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042828    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c76c09e9-3de9-4e99-964d-5a96bbfe54d2" (UID: "c76c09e9-3de9-4e99-964d-5a96bbfe54d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.042886 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c76c09e9-3de9-4e99-964d-5a96bbfe54d2" (UID: "c76c09e9-3de9-4e99-964d-5a96bbfe54d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.043209 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.042987    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.043209 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.043004    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.043209 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.043019    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.044797 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.044760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-kube-api-access-qh9bs" (OuterVolumeSpecName: "kube-api-access-qh9bs") pod "c76c09e9-3de9-4e99-964d-5a96bbfe54d2" (UID: "c76c09e9-3de9-4e99-964d-5a96bbfe54d2"). InnerVolumeSpecName "kube-api-access-qh9bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:16:31.044888 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.044807    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-dshm" (OuterVolumeSpecName: "dshm") pod "c76c09e9-3de9-4e99-964d-5a96bbfe54d2" (UID: "c76c09e9-3de9-4e99-964d-5a96bbfe54d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:16:31.044951 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.044934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c76c09e9-3de9-4e99-964d-5a96bbfe54d2" (UID: "c76c09e9-3de9-4e99-964d-5a96bbfe54d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:16:31.143990 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.143957    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.143990 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.143982    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.143990 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.143992    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9bs\" (UniqueName: \"kubernetes.io/projected/c76c09e9-3de9-4e99-964d-5a96bbfe54d2-kube-api-access-qh9bs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:16:31.453852 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.453821    2580 generic.go:356] "Generic (PLEG): container finished" podID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerID="d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2" exitCode=0
Jul 29 23:16:31.453852 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.453846    2580 generic.go:356] "Generic (PLEG): container finished" podID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerID="37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418" exitCode=0
Jul 29 23:16:31.454412 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.453871    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"
Jul 29 23:16:31.454412 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.453906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" event={"ID":"c76c09e9-3de9-4e99-964d-5a96bbfe54d2","Type":"ContainerDied","Data":"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"}
Jul 29 23:16:31.454412 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.453974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" event={"ID":"c76c09e9-3de9-4e99-964d-5a96bbfe54d2","Type":"ContainerDied","Data":"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"}
Jul 29 23:16:31.454412 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.454002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q" event={"ID":"c76c09e9-3de9-4e99-964d-5a96bbfe54d2","Type":"ContainerDied","Data":"9e3f79d6d89220e334d95fe9d2911b7c29c23cc3ce2b688472097b743ba81ff7"}
Jul 29 23:16:31.454412 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.454034    2580 scope.go:117] "RemoveContainer" containerID="d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"
Jul 29 23:16:31.463526 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.463509    2580 scope.go:117] "RemoveContainer" containerID="37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"
Jul 29 23:16:31.471705 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.471679    2580 scope.go:117] "RemoveContainer" containerID="d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"
Jul 29 23:16:31.472006 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:16:31.471985    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2\": container with ID starting with d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2 not found: ID does not exist" containerID="d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"
Jul 29 23:16:31.472057 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472015    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"} err="failed to get container status \"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2\": rpc error: code = NotFound desc = could not find container \"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2\": container with ID starting with d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2 not found: ID does not exist"
Jul 29 23:16:31.472057 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472034    2580 scope.go:117] "RemoveContainer" containerID="37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"
Jul 29 23:16:31.472318 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:16:31.472299    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418\": container with ID starting with 37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418 not found: ID does not exist" containerID="37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"
Jul 29 23:16:31.472379 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472326    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"} err="failed to get container status \"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418\": rpc error: code = NotFound desc = could not find container \"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418\": container with ID starting with 37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418 not found: ID does not exist"
Jul 29 23:16:31.472379 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472341    2580 scope.go:117] "RemoveContainer" containerID="d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"
Jul 29 23:16:31.472543 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472528    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2"} err="failed to get container status \"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2\": rpc error: code = NotFound desc = could not find container \"d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2\": container with ID starting with d8a717f1bc0cb2342f0bcb957a649f26de66ca06f462d1e67705ef617f07e1e2 not found: ID does not exist"
Jul 29 23:16:31.472595 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472543    2580 scope.go:117] "RemoveContainer" containerID="37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"
Jul 29 23:16:31.472771 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.472737    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418"} err="failed to get container status \"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418\": rpc error: code = NotFound desc = could not find container \"37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418\": container with ID starting with 37fbfb0ce6e54a1b14b4f9f7052e5c0ce64e8f440d4f534fa8ef9153fc19d418 not found: ID does not exist"
Jul 29 23:16:31.475859 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.475841    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"]
Jul 29 23:16:31.479557 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:31.479536    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-8mg7q"]
Jul 29 23:16:32.575166 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:32.575134    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" path="/var/lib/kubelet/pods/c76c09e9-3de9-4e99-964d-5a96bbfe54d2/volumes"
Jul 29 23:16:55.067132 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067097    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"]
Jul 29 23:16:55.067557 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="main"
Jul 29 23:16:55.067557 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067463    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="main"
Jul 29 23:16:55.067557 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067476    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="llm-d-routing-sidecar"
Jul 29 23:16:55.067557 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067481    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="llm-d-routing-sidecar"
Jul 29 23:16:55.067709 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067576    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="llm-d-routing-sidecar"
Jul 29 23:16:55.067709 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.067590    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76c09e9-3de9-4e99-964d-5a96bbfe54d2" containerName="main"
Jul 29 23:16:55.070818 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.070775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.072891 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.072867    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 29 23:16:55.073368 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.073352    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-bhfrc\""
Jul 29 23:16:55.073445 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.073412    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 29 23:16:55.073498 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.073470    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 29 23:16:55.089222 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.089198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"]
Jul 29 23:16:55.156842 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.156805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.157010 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.156940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9rz\" (UniqueName: \"kubernetes.io/projected/19d12f34-4429-4157-87ea-2eaf03e59ae0-kube-api-access-bx9rz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.157010 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.156989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d12f34-4429-4157-87ea-2eaf03e59ae0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.157108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.157016    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.157108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.157062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.157108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.157100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258053 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9rz\" (UniqueName: \"kubernetes.io/projected/19d12f34-4429-4157-87ea-2eaf03e59ae0-kube-api-access-bx9rz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258229 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d12f34-4429-4157-87ea-2eaf03e59ae0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258229 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258229 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258423 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258423 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258582 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258628 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.258663 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.258645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.260396 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.260375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.260676 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.260657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d12f34-4429-4157-87ea-2eaf03e59ae0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.266223 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.266201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9rz\" (UniqueName: \"kubernetes.io/projected/19d12f34-4429-4157-87ea-2eaf03e59ae0-kube-api-access-bx9rz\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.380750 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.380679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:16:55.514387 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.514361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"]
Jul 29 23:16:55.516049 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:16:55.516018    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d12f34_4429_4157_87ea_2eaf03e59ae0.slice/crio-0d3c3d241f4728550f1837e14bf5ea979851a228d10c2facf8f34539fe73db9e WatchSource:0}: Error finding container 0d3c3d241f4728550f1837e14bf5ea979851a228d10c2facf8f34539fe73db9e: Status 404 returned error can't find the container with id 0d3c3d241f4728550f1837e14bf5ea979851a228d10c2facf8f34539fe73db9e
Jul 29 23:16:55.589182 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:55.589157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" event={"ID":"19d12f34-4429-4157-87ea-2eaf03e59ae0","Type":"ContainerStarted","Data":"0d3c3d241f4728550f1837e14bf5ea979851a228d10c2facf8f34539fe73db9e"}
Jul 29 23:16:56.595969 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:56.595931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" event={"ID":"19d12f34-4429-4157-87ea-2eaf03e59ae0","Type":"ContainerStarted","Data":"5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"}
Jul 29 23:16:56.615286 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:56.615237    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" podStartSLOduration=1.615221279 podStartE2EDuration="1.615221279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:16:56.612586106 +0000 UTC m=+1832.521276465" watchObservedRunningTime="2026-07-29 23:16:56.615221279 +0000 UTC m=+1832.523911654"
Jul 29 23:16:57.381108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:16:57.381073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:17:05.381560 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:05.381519    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:17:05.394264 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:05.394237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:17:05.664253 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:05.664171    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:17:28.889532 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:28.889488    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"]
Jul 29 23:17:28.898744 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:17:28.898703    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:17:28.898744 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:17:28.898744 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:17:28.898744 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb" cmd=["/bin/sleep","15"]
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:28.898757    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" message="" err=<
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:17:28.898770    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" podUID="19d12f34-4429-4157-87ea-2eaf03e59ae0" containerName="main" containerID="cri-o://5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"
Jul 29 23:17:28.899094 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:28.898827    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" podUID="19d12f34-4429-4157-87ea-2eaf03e59ae0" containerName="main" containerID="cri-o://5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb" gracePeriod=60
Jul 29 23:17:29.148490 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.148435    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:17:29.244621 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.244585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx9rz\" (UniqueName: \"kubernetes.io/projected/19d12f34-4429-4157-87ea-2eaf03e59ae0-kube-api-access-bx9rz\") pod \"19d12f34-4429-4157-87ea-2eaf03e59ae0\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") "
Jul 29 23:17:29.244815 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.244632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-dshm\") pod \"19d12f34-4429-4157-87ea-2eaf03e59ae0\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") "
Jul 29 23:17:29.244815 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.244654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-model-cache\") pod \"19d12f34-4429-4157-87ea-2eaf03e59ae0\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") "
Jul 29 23:17:29.244815 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.244690    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-tmp-dir\") pod \"19d12f34-4429-4157-87ea-2eaf03e59ae0\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") "
Jul 29 23:17:29.244815 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.244723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d12f34-4429-4157-87ea-2eaf03e59ae0-tls-certs\") pod \"19d12f34-4429-4157-87ea-2eaf03e59ae0\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") "
Jul 29 23:17:29.244815 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.244756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-home\") pod \"19d12f34-4429-4157-87ea-2eaf03e59ae0\" (UID: \"19d12f34-4429-4157-87ea-2eaf03e59ae0\") "
Jul 29 23:17:29.245102 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.245003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-model-cache" (OuterVolumeSpecName: "model-cache") pod "19d12f34-4429-4157-87ea-2eaf03e59ae0" (UID: "19d12f34-4429-4157-87ea-2eaf03e59ae0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.245102 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.245029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19d12f34-4429-4157-87ea-2eaf03e59ae0" (UID: "19d12f34-4429-4157-87ea-2eaf03e59ae0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.245205 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.245110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-home" (OuterVolumeSpecName: "home") pod "19d12f34-4429-4157-87ea-2eaf03e59ae0" (UID: "19d12f34-4429-4157-87ea-2eaf03e59ae0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.247043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.247019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-dshm" (OuterVolumeSpecName: "dshm") pod "19d12f34-4429-4157-87ea-2eaf03e59ae0" (UID: "19d12f34-4429-4157-87ea-2eaf03e59ae0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:17:29.247043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.247024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d12f34-4429-4157-87ea-2eaf03e59ae0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19d12f34-4429-4157-87ea-2eaf03e59ae0" (UID: "19d12f34-4429-4157-87ea-2eaf03e59ae0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:17:29.247187 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.247049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d12f34-4429-4157-87ea-2eaf03e59ae0-kube-api-access-bx9rz" (OuterVolumeSpecName: "kube-api-access-bx9rz") pod "19d12f34-4429-4157-87ea-2eaf03e59ae0" (UID: "19d12f34-4429-4157-87ea-2eaf03e59ae0"). InnerVolumeSpecName "kube-api-access-bx9rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:17:29.346276 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.346238    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.346276 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.346268    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19d12f34-4429-4157-87ea-2eaf03e59ae0-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.346276 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.346276    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.346276 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.346285    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9rz\" (UniqueName: \"kubernetes.io/projected/19d12f34-4429-4157-87ea-2eaf03e59ae0-kube-api-access-bx9rz\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.346552 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.346293    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.346552 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.346302    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19d12f34-4429-4157-87ea-2eaf03e59ae0-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:17:29.787668 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.787635    2580 generic.go:356] "Generic (PLEG): container finished" podID="19d12f34-4429-4157-87ea-2eaf03e59ae0" containerID="5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb" exitCode=0
Jul 29 23:17:29.787890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.787680    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"
Jul 29 23:17:29.787890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.787720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" event={"ID":"19d12f34-4429-4157-87ea-2eaf03e59ae0","Type":"ContainerDied","Data":"5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"}
Jul 29 23:17:29.787890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.787803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2" event={"ID":"19d12f34-4429-4157-87ea-2eaf03e59ae0","Type":"ContainerDied","Data":"0d3c3d241f4728550f1837e14bf5ea979851a228d10c2facf8f34539fe73db9e"}
Jul 29 23:17:29.787890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.787837    2580 scope.go:117] "RemoveContainer" containerID="5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"
Jul 29 23:17:29.798191 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.798172    2580 scope.go:117] "RemoveContainer" containerID="5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"
Jul 29 23:17:29.798438 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:17:29.798419    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb\": container with ID starting with 5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb not found: ID does not exist" containerID="5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"
Jul 29 23:17:29.798510 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.798446    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb"} err="failed to get container status \"5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb\": rpc error: code = NotFound desc = could not find container \"5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb\": container with ID starting with 5f4fd4a82c066ceda51da76f310cfe661f490a3595c8f13a1d876ef7d4849adb not found: ID does not exist"
Jul 29 23:17:29.809247 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.809222    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"]
Jul 29 23:17:29.812834 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:29.812814    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58ftwfw2"]
Jul 29 23:17:30.569567 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:17:30.569523    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d12f34-4429-4157-87ea-2eaf03e59ae0" path="/var/lib/kubelet/pods/19d12f34-4429-4157-87ea-2eaf03e59ae0/volumes"
Jul 29 23:18:04.611105 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.611065    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"]
Jul 29 23:18:04.611580 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.611452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d12f34-4429-4157-87ea-2eaf03e59ae0" containerName="main"
Jul 29 23:18:04.611580 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.611464    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d12f34-4429-4157-87ea-2eaf03e59ae0" containerName="main"
Jul 29 23:18:04.611580 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.611561    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d12f34-4429-4157-87ea-2eaf03e59ae0" containerName="main"
Jul 29 23:18:04.614469 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.614449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.616804 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.616768    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 29 23:18:04.616937 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.616858    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 29 23:18:04.617251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.617234    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 29 23:18:04.617301 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.617244    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 29 23:18:04.617403 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.617384    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rtb5v\""
Jul 29 23:18:04.626635 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.626611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"]
Jul 29 23:18:04.760179 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-model-cache\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.760179 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.760405 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-dshm\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.760405 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.760405 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r65l\" (UniqueName: \"kubernetes.io/projected/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kube-api-access-4r65l\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.760405 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-home\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.760405 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.760370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861522 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-home\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861522 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861500    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-model-cache\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861584    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-dshm\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.861773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r65l\" (UniqueName: \"kubernetes.io/projected/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kube-api-access-4r65l\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.862087 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-home\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.862087 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.861880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.862202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.862133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-model-cache\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.862202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.862156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.864249 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.864230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-dshm\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.864407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.864388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.877114 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.876767    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"]
Jul 29 23:18:04.877114 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.877039    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r65l\" (UniqueName: \"kubernetes.io/projected/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kube-api-access-4r65l\") pod \"tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:04.880899 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.880874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:04.892044 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.891978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"]
Jul 29 23:18:04.926818 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:04.926777    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:05.063857 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.063826    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.064024 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.063867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.064024 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.063891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9261d3a8-ef39-44a0-8bb1-40f39b880533-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.070837 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.070802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"]
Jul 29 23:18:05.073846 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:18:05.073818    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e45fc0d_fa59_46eb_9c8e_fbd24871568a.slice/crio-3e88545ab2e880ba4d5080b45ce8e026f51f89d71e8f28184164d04378aecead WatchSource:0}: Error finding container 3e88545ab2e880ba4d5080b45ce8e026f51f89d71e8f28184164d04378aecead: Status 404 returned error can't find the container with id 3e88545ab2e880ba4d5080b45ce8e026f51f89d71e8f28184164d04378aecead
Jul 29 23:18:05.165032 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.164996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.165032 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.165036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.165247 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.165059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9261d3a8-ef39-44a0-8bb1-40f39b880533-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.165466 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.165441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.165503 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.165459    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.167549 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.167529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9261d3a8-ef39-44a0-8bb1-40f39b880533-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.208994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.208958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:05.351369 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.351339    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"]
Jul 29 23:18:05.353457 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:18:05.353429    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9261d3a8_ef39_44a0_8bb1_40f39b880533.slice/crio-57aff1bb0946c30789c3c864c21fae8e78cefb179faa9ddf2f1f35979d824523 WatchSource:0}: Error finding container 57aff1bb0946c30789c3c864c21fae8e78cefb179faa9ddf2f1f35979d824523: Status 404 returned error can't find the container with id 57aff1bb0946c30789c3c864c21fae8e78cefb179faa9ddf2f1f35979d824523
Jul 29 23:18:05.979471 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.979436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" event={"ID":"7e45fc0d-fa59-46eb-9c8e-fbd24871568a","Type":"ContainerStarted","Data":"f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40"}
Jul 29 23:18:05.980004 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.979501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" event={"ID":"7e45fc0d-fa59-46eb-9c8e-fbd24871568a","Type":"ContainerStarted","Data":"3e88545ab2e880ba4d5080b45ce8e026f51f89d71e8f28184164d04378aecead"}
Jul 29 23:18:05.981005 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.980974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" event={"ID":"9261d3a8-ef39-44a0-8bb1-40f39b880533","Type":"ContainerStarted","Data":"9aabe2db1cb6f77ab5e4cb8233721d5aba35f37d96526093c85912c6fc26b375"}
Jul 29 23:18:05.981141 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:05.981032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" event={"ID":"9261d3a8-ef39-44a0-8bb1-40f39b880533","Type":"ContainerStarted","Data":"57aff1bb0946c30789c3c864c21fae8e78cefb179faa9ddf2f1f35979d824523"}
Jul 29 23:18:11.016994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:11.016960    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerID="f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40" exitCode=0
Jul 29 23:18:11.017478 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:11.017033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" event={"ID":"7e45fc0d-fa59-46eb-9c8e-fbd24871568a","Type":"ContainerDied","Data":"f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40"}
Jul 29 23:18:11.018830 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:11.018777    2580 generic.go:356] "Generic (PLEG): container finished" podID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerID="9aabe2db1cb6f77ab5e4cb8233721d5aba35f37d96526093c85912c6fc26b375" exitCode=0
Jul 29 23:18:11.018830 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:11.018812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" event={"ID":"9261d3a8-ef39-44a0-8bb1-40f39b880533","Type":"ContainerDied","Data":"9aabe2db1cb6f77ab5e4cb8233721d5aba35f37d96526093c85912c6fc26b375"}
Jul 29 23:18:11.019884 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:11.019863    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 23:18:12.030007 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:12.029962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" event={"ID":"7e45fc0d-fa59-46eb-9c8e-fbd24871568a","Type":"ContainerStarted","Data":"74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"}
Jul 29 23:18:12.048455 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:12.048391    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" podStartSLOduration=2.873251239 podStartE2EDuration="8.048369504s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.175118265s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:18:12.047447184 +0000 UTC m=+1907.956137543" watchObservedRunningTime="2026-07-29 23:18:12.048369504 +0000 UTC m=+1907.957059863"
Jul 29 23:18:12.927251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:12.927198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:14.927402 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:14.927364    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:14.942459 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:14.942432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:15.064892 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:15.064859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:18:49.267539 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:49.267453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" event={"ID":"9261d3a8-ef39-44a0-8bb1-40f39b880533","Type":"ContainerStarted","Data":"015fe1b30310ca70cfa43e9c455435575ac46b24f85fd8436beb87a0bf216742"}
Jul 29 23:18:49.268014 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:49.267671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:18:49.291860 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:18:49.291782    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" podStartSLOduration=2.716671792 podStartE2EDuration="45.291758429s" totalImagesPullingTime="37.898239182s" totalInitContainerRuntime="4.676847455s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:18:49.290913058 +0000 UTC m=+1945.199603415" watchObservedRunningTime="2026-07-29 23:18:49.291758429 +0000 UTC m=+1945.200448788"
Jul 29 23:19:20.284835 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:20.284718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:19:21.341646 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:21.341614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4_9261d3a8-ef39-44a0-8bb1-40f39b880533/main/0.log"
Jul 29 23:19:21.817388 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:21.817355    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"]
Jul 29 23:19:21.817601 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:21.817577    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerName="main" containerID="cri-o://015fe1b30310ca70cfa43e9c455435575ac46b24f85fd8436beb87a0bf216742" gracePeriod=30
Jul 29 23:19:21.828777 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:21.828744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"]
Jul 29 23:19:21.843604 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:19:21.843562    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:19:21.843604 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:19:21.843604 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:19:21.843604 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679" cmd=["/bin/sleep","15"]
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:21.843626    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" message="" err=<
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:19:21.843643    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="main" containerID="cri-o://74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"
Jul 29 23:19:21.843969 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:21.843685    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="main" containerID="cri-o://74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679" gracePeriod=60
Jul 29 23:19:22.051160 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.051115    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 29 23:19:22.304447 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.304423    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:19:22.414412 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tmp-dir\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-model-cache\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414476    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-dshm\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-home\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414557    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r65l\" (UniqueName: \"kubernetes.io/projected/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kube-api-access-4r65l\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tls-certs\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kserve-provision-location\") pod \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\" (UID: \"7e45fc0d-fa59-46eb-9c8e-fbd24871568a\") "
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414724    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.414872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.414829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-home" (OuterVolumeSpecName: "home") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.415462 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.415128    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.415462 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.415151    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.415462 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.415166    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.417040 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.417019    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-dshm" (OuterVolumeSpecName: "dshm") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.417181 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.417144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kube-api-access-4r65l" (OuterVolumeSpecName: "kube-api-access-4r65l") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "kube-api-access-4r65l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:19:22.417240 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.417191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:19:22.451649 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.451611    2580 generic.go:356] "Generic (PLEG): container finished" podID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerID="74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679" exitCode=0
Jul 29 23:19:22.451847 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.451661    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"
Jul 29 23:19:22.451847 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.451697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" event={"ID":"7e45fc0d-fa59-46eb-9c8e-fbd24871568a","Type":"ContainerDied","Data":"74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"}
Jul 29 23:19:22.451847 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.451753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9" event={"ID":"7e45fc0d-fa59-46eb-9c8e-fbd24871568a","Type":"ContainerDied","Data":"3e88545ab2e880ba4d5080b45ce8e026f51f89d71e8f28184164d04378aecead"}
Jul 29 23:19:22.451847 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.451810    2580 scope.go:117] "RemoveContainer" containerID="74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"
Jul 29 23:19:22.453780 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.453753    2580 generic.go:356] "Generic (PLEG): container finished" podID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerID="015fe1b30310ca70cfa43e9c455435575ac46b24f85fd8436beb87a0bf216742" exitCode=0
Jul 29 23:19:22.453904 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.453846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" event={"ID":"9261d3a8-ef39-44a0-8bb1-40f39b880533","Type":"ContainerDied","Data":"015fe1b30310ca70cfa43e9c455435575ac46b24f85fd8436beb87a0bf216742"}
Jul 29 23:19:22.462260 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.462237    2580 scope.go:117] "RemoveContainer" containerID="f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40"
Jul 29 23:19:22.464620 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.464602    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:19:22.472253 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.472233    2580 scope.go:117] "RemoveContainer" containerID="74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"
Jul 29 23:19:22.472519 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:19:22.472501    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679\": container with ID starting with 74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679 not found: ID does not exist" containerID="74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"
Jul 29 23:19:22.472578 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.472527    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679"} err="failed to get container status \"74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679\": rpc error: code = NotFound desc = could not find container \"74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679\": container with ID starting with 74e476adaef859e4f58b8b59eb37baec6d56fb37726ec69dd1b58956d5de2679 not found: ID does not exist"
Jul 29 23:19:22.472578 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.472545    2580 scope.go:117] "RemoveContainer" containerID="f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40"
Jul 29 23:19:22.472805 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:19:22.472770    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40\": container with ID starting with f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40 not found: ID does not exist" containerID="f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40"
Jul 29 23:19:22.472858 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.472805    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40"} err="failed to get container status \"f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40\": rpc error: code = NotFound desc = could not find container \"f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40\": container with ID starting with f212f13becd9a2983b293f49a2ecb85e885b867e69ddc130fe5126558f53cd40 not found: ID does not exist"
Jul 29 23:19:22.484488 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.484455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e45fc0d-fa59-46eb-9c8e-fbd24871568a" (UID: "7e45fc0d-fa59-46eb-9c8e-fbd24871568a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.515573 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.515541    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.515573 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.515567    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r65l\" (UniqueName: \"kubernetes.io/projected/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kube-api-access-4r65l\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.515573 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.515577    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.515860 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.515588    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e45fc0d-fa59-46eb-9c8e-fbd24871568a-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.616265 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.616231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9261d3a8-ef39-44a0-8bb1-40f39b880533-tls-certs\") pod \"9261d3a8-ef39-44a0-8bb1-40f39b880533\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") "
Jul 29 23:19:22.616473 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.616286    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-kserve-provision-location\") pod \"9261d3a8-ef39-44a0-8bb1-40f39b880533\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") "
Jul 29 23:19:22.616473 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.616333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-tmp-dir\") pod \"9261d3a8-ef39-44a0-8bb1-40f39b880533\" (UID: \"9261d3a8-ef39-44a0-8bb1-40f39b880533\") "
Jul 29 23:19:22.616709 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.616679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9261d3a8-ef39-44a0-8bb1-40f39b880533" (UID: "9261d3a8-ef39-44a0-8bb1-40f39b880533"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.618246 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.618225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9261d3a8-ef39-44a0-8bb1-40f39b880533-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9261d3a8-ef39-44a0-8bb1-40f39b880533" (UID: "9261d3a8-ef39-44a0-8bb1-40f39b880533"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:19:22.674107 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.674008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9261d3a8-ef39-44a0-8bb1-40f39b880533" (UID: "9261d3a8-ef39-44a0-8bb1-40f39b880533"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:19:22.718023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.717994    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9261d3a8-ef39-44a0-8bb1-40f39b880533-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.718023 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.718021    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.718202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.718031    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9261d3a8-ef39-44a0-8bb1-40f39b880533-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:19:22.771283 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.771252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"]
Jul 29 23:19:22.779990 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:22.779952    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-9985ff8f4-hrmh9"]
Jul 29 23:19:23.461317 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:23.461292    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"
Jul 29 23:19:23.461317 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:23.461300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4" event={"ID":"9261d3a8-ef39-44a0-8bb1-40f39b880533","Type":"ContainerDied","Data":"57aff1bb0946c30789c3c864c21fae8e78cefb179faa9ddf2f1f35979d824523"}
Jul 29 23:19:23.461855 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:23.461355    2580 scope.go:117] "RemoveContainer" containerID="015fe1b30310ca70cfa43e9c455435575ac46b24f85fd8436beb87a0bf216742"
Jul 29 23:19:23.471269 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:23.471253    2580 scope.go:117] "RemoveContainer" containerID="9aabe2db1cb6f77ab5e4cb8233721d5aba35f37d96526093c85912c6fc26b375"
Jul 29 23:19:23.483425 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:23.483404    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"]
Jul 29 23:19:23.488505 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:23.488481    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-598b4f8555-k7hz4"]
Jul 29 23:19:24.569660 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:24.569628    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" path="/var/lib/kubelet/pods/7e45fc0d-fa59-46eb-9c8e-fbd24871568a/volumes"
Jul 29 23:19:24.570056 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:24.570041    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" path="/var/lib/kubelet/pods/9261d3a8-ef39-44a0-8bb1-40f39b880533/volumes"
Jul 29 23:19:54.815380 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815337    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"]
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815828    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerName="main"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815844    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerName="main"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815862    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="storage-initializer"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="storage-initializer"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815878    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerName="storage-initializer"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815883    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerName="storage-initializer"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815895    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="main"
Jul 29 23:19:54.815913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815902    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="main"
Jul 29 23:19:54.816211 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815964    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e45fc0d-fa59-46eb-9c8e-fbd24871568a" containerName="main"
Jul 29 23:19:54.816211 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.815974    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9261d3a8-ef39-44a0-8bb1-40f39b880533" containerName="main"
Jul 29 23:19:54.818227 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.818212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.820350 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.820326    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 29 23:19:54.820511 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.820487    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 29 23:19:54.820629 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.820517    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 29 23:19:54.820629 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.820436    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9lq75\""
Jul 29 23:19:54.820629 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.820459    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 29 23:19:54.831349 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.831319    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"]
Jul 29 23:19:54.891560 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.891729 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-dshm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.891729 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.891729 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-home\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.891729 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-model-cache\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.891729 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cncm\" (UniqueName: \"kubernetes.io/projected/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kube-api-access-5cncm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.891926 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.891737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993055 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-dshm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-home\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-model-cache\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cncm\" (UniqueName: \"kubernetes.io/projected/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kube-api-access-5cncm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993655 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993628    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993736 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-model-cache\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993736 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.993736 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.993719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-home\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.995474 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.995455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-dshm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:54.995636 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:54.995620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:55.001695 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.001672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cncm\" (UniqueName: \"kubernetes.io/projected/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kube-api-access-5cncm\") pod \"tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:55.084084 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.084010    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"]
Jul 29 23:19:55.087002 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.086985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.102370 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.102334    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"]
Jul 29 23:19:55.128511 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.128476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:19:55.194694 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.194649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7aed04-6de2-408b-9828-fea004b36d02-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.195046 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.194702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.195046 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.194861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.267881 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.267847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"]
Jul 29 23:19:55.270064 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:19:55.270032    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd4d295_e8f5_417a_91c6_63aa2031fbea.slice/crio-de53b7baf3359e4a552c881fd1fb53abf538e2838e8757b96748d0482299aab9 WatchSource:0}: Error finding container de53b7baf3359e4a552c881fd1fb53abf538e2838e8757b96748d0482299aab9: Status 404 returned error can't find the container with id de53b7baf3359e4a552c881fd1fb53abf538e2838e8757b96748d0482299aab9
Jul 29 23:19:55.296502 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.296471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.296607 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.296535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7aed04-6de2-408b-9828-fea004b36d02-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.296607 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.296553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.296851 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.296831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.296894 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.296853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.298810 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.298777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7aed04-6de2-408b-9828-fea004b36d02-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-759567958f-zwkgp\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.397261 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.397176    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:19:55.539324 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.539297    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"]
Jul 29 23:19:55.541368 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:19:55.541344    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7aed04_6de2_408b_9828_fea004b36d02.slice/crio-4764e155d2b1beefab7877d2aa9bba20a58f200dd078e14870ec08ed1b0b4585 WatchSource:0}: Error finding container 4764e155d2b1beefab7877d2aa9bba20a58f200dd078e14870ec08ed1b0b4585: Status 404 returned error can't find the container with id 4764e155d2b1beefab7877d2aa9bba20a58f200dd078e14870ec08ed1b0b4585
Jul 29 23:19:55.634544 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.634509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" event={"ID":"5dd4d295-e8f5-417a-91c6-63aa2031fbea","Type":"ContainerStarted","Data":"3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0"}
Jul 29 23:19:55.634721 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.634573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" event={"ID":"5dd4d295-e8f5-417a-91c6-63aa2031fbea","Type":"ContainerStarted","Data":"de53b7baf3359e4a552c881fd1fb53abf538e2838e8757b96748d0482299aab9"}
Jul 29 23:19:55.636299 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.636273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" event={"ID":"bd7aed04-6de2-408b-9828-fea004b36d02","Type":"ContainerStarted","Data":"ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661"}
Jul 29 23:19:55.636408 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:19:55.636320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" event={"ID":"bd7aed04-6de2-408b-9828-fea004b36d02","Type":"ContainerStarted","Data":"4764e155d2b1beefab7877d2aa9bba20a58f200dd078e14870ec08ed1b0b4585"}
Jul 29 23:20:00.669011 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:00.668978    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd7aed04-6de2-408b-9828-fea004b36d02" containerID="ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661" exitCode=0
Jul 29 23:20:00.669456 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:00.669058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" event={"ID":"bd7aed04-6de2-408b-9828-fea004b36d02","Type":"ContainerDied","Data":"ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661"}
Jul 29 23:20:00.670733 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:00.670707    2580 generic.go:356] "Generic (PLEG): container finished" podID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerID="3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0" exitCode=0
Jul 29 23:20:00.670839 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:00.670778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" event={"ID":"5dd4d295-e8f5-417a-91c6-63aa2031fbea","Type":"ContainerDied","Data":"3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0"}
Jul 29 23:20:01.678542 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:01.678504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" event={"ID":"bd7aed04-6de2-408b-9828-fea004b36d02","Type":"ContainerStarted","Data":"11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e"}
Jul 29 23:20:01.679016 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:01.678745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:20:01.680571 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:01.680547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" event={"ID":"5dd4d295-e8f5-417a-91c6-63aa2031fbea","Type":"ContainerStarted","Data":"bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"}
Jul 29 23:20:01.696119 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:01.696074    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" podStartSLOduration=2.027182128 podStartE2EDuration="6.696060056s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.668877928s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:20:01.694688675 +0000 UTC m=+2017.603379033" watchObservedRunningTime="2026-07-29 23:20:01.696060056 +0000 UTC m=+2017.604750414"
Jul 29 23:20:01.711983 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:01.711909    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" podStartSLOduration=2.705720902 podStartE2EDuration="7.711890137s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.006169235s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:20:01.710639684 +0000 UTC m=+2017.619330045" watchObservedRunningTime="2026-07-29 23:20:01.711890137 +0000 UTC m=+2017.620580499"
Jul 29 23:20:02.129194 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:02.129145    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:20:05.128605 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:05.128568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:20:05.147263 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:05.147231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:20:05.714363 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:05.714317    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:20:32.698270 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:32.698237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:20:33.725135 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:33.725090    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-759567958f-zwkgp_bd7aed04-6de2-408b-9828-fea004b36d02/main/0.log"
Jul 29 23:20:33.962386 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:33.962357    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"]
Jul 29 23:20:33.962656 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:33.962604    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" containerName="main" containerID="cri-o://11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e" gracePeriod=30
Jul 29 23:20:33.965257 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:33.965231    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"]
Jul 29 23:20:33.974011 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:20:33.973978    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:20:33.974011 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:20:33.974011 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:20:33.974011 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c" cmd=["/bin/sleep","15"]
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:33.974030    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" message="" err=<
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:20:33.974043    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="main" containerID="cri-o://bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"
Jul 29 23:20:33.974307 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:33.974079    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="main" containerID="cri-o://bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c" gracePeriod=60
Jul 29 23:20:34.341378 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.341297    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:20:34.358778 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.358750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-dshm\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.358915 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.358828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tls-certs\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.358915 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.358876    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cncm\" (UniqueName: \"kubernetes.io/projected/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kube-api-access-5cncm\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.359027 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.358941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kserve-provision-location\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.359083 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-home\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.359137 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359106    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tmp-dir\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.359137 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359127    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-model-cache\") pod \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\" (UID: \"5dd4d295-e8f5-417a-91c6-63aa2031fbea\") "
Jul 29 23:20:34.359313 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-home" (OuterVolumeSpecName: "home") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.359464 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359444    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.359620 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-model-cache" (OuterVolumeSpecName: "model-cache") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.359735 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.359706    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.361204 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.361174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:20:34.361464 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.361433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kube-api-access-5cncm" (OuterVolumeSpecName: "kube-api-access-5cncm") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "kube-api-access-5cncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:20:34.361565 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.361536    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-dshm" (OuterVolumeSpecName: "dshm") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.432007 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.431925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5dd4d295-e8f5-417a-91c6-63aa2031fbea" (UID: "5dd4d295-e8f5-417a-91c6-63aa2031fbea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.460876 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.460841    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.460876 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.460877    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.461043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.460893    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.461043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.460905    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd4d295-e8f5-417a-91c6-63aa2031fbea-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.461043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.460918    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd4d295-e8f5-417a-91c6-63aa2031fbea-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.461043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.460941    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cncm\" (UniqueName: \"kubernetes.io/projected/5dd4d295-e8f5-417a-91c6-63aa2031fbea-kube-api-access-5cncm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.532973 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.532947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:20:34.561281 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.561252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-tmp-dir\") pod \"bd7aed04-6de2-408b-9828-fea004b36d02\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") "
Jul 29 23:20:34.561451 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.561299    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7aed04-6de2-408b-9828-fea004b36d02-tls-certs\") pod \"bd7aed04-6de2-408b-9828-fea004b36d02\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") "
Jul 29 23:20:34.561451 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.561335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-kserve-provision-location\") pod \"bd7aed04-6de2-408b-9828-fea004b36d02\" (UID: \"bd7aed04-6de2-408b-9828-fea004b36d02\") "
Jul 29 23:20:34.561633 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.561609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd7aed04-6de2-408b-9828-fea004b36d02" (UID: "bd7aed04-6de2-408b-9828-fea004b36d02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.563547 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.563513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7aed04-6de2-408b-9828-fea004b36d02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd7aed04-6de2-408b-9828-fea004b36d02" (UID: "bd7aed04-6de2-408b-9828-fea004b36d02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:20:34.620483 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.620394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd7aed04-6de2-408b-9828-fea004b36d02" (UID: "bd7aed04-6de2-408b-9828-fea004b36d02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:20:34.662594 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.662563    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.662758 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.662666    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd7aed04-6de2-408b-9828-fea004b36d02-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.662758 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.662688    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7aed04-6de2-408b-9828-fea004b36d02-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:20:34.869326 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.869291    2580 generic.go:356] "Generic (PLEG): container finished" podID="bd7aed04-6de2-408b-9828-fea004b36d02" containerID="11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e" exitCode=0
Jul 29 23:20:34.869828 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.869364    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"
Jul 29 23:20:34.869828 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.869380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" event={"ID":"bd7aed04-6de2-408b-9828-fea004b36d02","Type":"ContainerDied","Data":"11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e"}
Jul 29 23:20:34.869828 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.869465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp" event={"ID":"bd7aed04-6de2-408b-9828-fea004b36d02","Type":"ContainerDied","Data":"4764e155d2b1beefab7877d2aa9bba20a58f200dd078e14870ec08ed1b0b4585"}
Jul 29 23:20:34.869828 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.869498    2580 scope.go:117] "RemoveContainer" containerID="11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e"
Jul 29 23:20:34.871253 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.871230    2580 generic.go:356] "Generic (PLEG): container finished" podID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerID="bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c" exitCode=0
Jul 29 23:20:34.871352 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.871290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"
Jul 29 23:20:34.871352 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.871300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" event={"ID":"5dd4d295-e8f5-417a-91c6-63aa2031fbea","Type":"ContainerDied","Data":"bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"}
Jul 29 23:20:34.871460 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.871383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2" event={"ID":"5dd4d295-e8f5-417a-91c6-63aa2031fbea","Type":"ContainerDied","Data":"de53b7baf3359e4a552c881fd1fb53abf538e2838e8757b96748d0482299aab9"}
Jul 29 23:20:34.879111 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.878944    2580 scope.go:117] "RemoveContainer" containerID="ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661"
Jul 29 23:20:34.887687 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.887618    2580 scope.go:117] "RemoveContainer" containerID="11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e"
Jul 29 23:20:34.888062 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:20:34.888033    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e\": container with ID starting with 11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e not found: ID does not exist" containerID="11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e"
Jul 29 23:20:34.888154 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.888072    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e"} err="failed to get container status \"11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e\": rpc error: code = NotFound desc = could not find container \"11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e\": container with ID starting with 11b912d8a75560caa9ca59323c8cc67c2db5970d3d8e3176261becf5f8dd226e not found: ID does not exist"
Jul 29 23:20:34.888154 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.888096    2580 scope.go:117] "RemoveContainer" containerID="ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661"
Jul 29 23:20:34.888370 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:20:34.888353    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661\": container with ID starting with ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661 not found: ID does not exist" containerID="ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661"
Jul 29 23:20:34.888409 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.888378    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661"} err="failed to get container status \"ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661\": rpc error: code = NotFound desc = could not find container \"ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661\": container with ID starting with ecf90d308e589aab3e659778e44263656a8b01649d9b97608211699f03051661 not found: ID does not exist"
Jul 29 23:20:34.888409 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.888393    2580 scope.go:117] "RemoveContainer" containerID="bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"
Jul 29 23:20:34.890557 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.890537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"]
Jul 29 23:20:34.896715 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.896694    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5c558c7c55-k6jh2"]
Jul 29 23:20:34.898576 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.898557    2580 scope.go:117] "RemoveContainer" containerID="3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0"
Jul 29 23:20:34.906336 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.906302    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"]
Jul 29 23:20:34.908367 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.908348    2580 scope.go:117] "RemoveContainer" containerID="bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"
Jul 29 23:20:34.908608 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:20:34.908589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c\": container with ID starting with bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c not found: ID does not exist" containerID="bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"
Jul 29 23:20:34.908690 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.908613    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c"} err="failed to get container status \"bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c\": rpc error: code = NotFound desc = could not find container \"bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c\": container with ID starting with bf05ff50fc51cabe39818d44ec4caaa7dd7b064731e04daa3d1b7999e357569c not found: ID does not exist"
Jul 29 23:20:34.908690 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.908631    2580 scope.go:117] "RemoveContainer" containerID="3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0"
Jul 29 23:20:34.908885 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:20:34.908862    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0\": container with ID starting with 3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0 not found: ID does not exist" containerID="3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0"
Jul 29 23:20:34.908953 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.908890    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0"} err="failed to get container status \"3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0\": rpc error: code = NotFound desc = could not find container \"3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0\": container with ID starting with 3c353e6615814a7344cb51e57586daac21fe5d41d37c8c98b70a054d0681c9a0 not found: ID does not exist"
Jul 29 23:20:34.910249 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:34.910230    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-759567958f-zwkgp"]
Jul 29 23:20:36.569609 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:36.569576    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" path="/var/lib/kubelet/pods/5dd4d295-e8f5-417a-91c6-63aa2031fbea/volumes"
Jul 29 23:20:36.570094 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:36.569994    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" path="/var/lib/kubelet/pods/bd7aed04-6de2-408b-9828-fea004b36d02/volumes"
Jul 29 23:20:47.688824 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:47.688717    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-c6vx2" podUID="8728adf3-07a6-4869-8405-6e7ba6a7ec6f" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 23:20:47.689196 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:47.688885    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 23:20:47.689613 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:47.689585    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-c6vx2" podUID="8728adf3-07a6-4869-8405-6e7ba6a7ec6f" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 23:20:47.942830 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:47.942722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c6vx2_8728adf3-07a6-4869-8405-6e7ba6a7ec6f/konnectivity-agent/0.log" limit=1024
Jul 29 23:20:59.277811 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.277758    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"]
Jul 29 23:20:59.278380 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278357    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="main"
Jul 29 23:20:59.278380 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278381    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="main"
Jul 29 23:20:59.278562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278396    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="storage-initializer"
Jul 29 23:20:59.278562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278404    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="storage-initializer"
Jul 29 23:20:59.278562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278418    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" containerName="storage-initializer"
Jul 29 23:20:59.278562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278426    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" containerName="storage-initializer"
Jul 29 23:20:59.278562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278450    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" containerName="main"
Jul 29 23:20:59.278562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278458    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" containerName="main"
Jul 29 23:20:59.278904 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278585    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7aed04-6de2-408b-9828-fea004b36d02" containerName="main"
Jul 29 23:20:59.278904 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.278604    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd4d295-e8f5-417a-91c6-63aa2031fbea" containerName="main"
Jul 29 23:20:59.280523 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.280495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.282568 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.282546    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 29 23:20:59.283089 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.283071    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 29 23:20:59.283089 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.283096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 29 23:20:59.283264 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.283173    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-v2sjp\""
Jul 29 23:20:59.294007 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.293968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"]
Jul 29 23:20:59.377340 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.377307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.377493 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.377343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.377493 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.377419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.377493 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.377484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.377616 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.377511    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.377616 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.377532    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hslfh\" (UniqueName: \"kubernetes.io/projected/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-kube-api-access-hslfh\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.478978 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.478946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.478978 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.478983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hslfh\" (UniqueName: \"kubernetes.io/projected/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-kube-api-access-hslfh\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479202 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479527 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479593 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.479634 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.479583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.481311 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.481286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.481563 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.481543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.489453 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.489430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hslfh\" (UniqueName: \"kubernetes.io/projected/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-kube-api-access-hslfh\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.597095 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.597063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:20:59.729696 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:20:59.729669    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"]
Jul 29 23:20:59.733105 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:20:59.733078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b50d2b_c8ee_4672_9e45_4a4d9d64ae3e.slice/crio-d31877688dc6a731ab84c1354f4e290b913a5b727077988162df34b5f8f7cd01 WatchSource:0}: Error finding container d31877688dc6a731ab84c1354f4e290b913a5b727077988162df34b5f8f7cd01: Status 404 returned error can't find the container with id d31877688dc6a731ab84c1354f4e290b913a5b727077988162df34b5f8f7cd01
Jul 29 23:21:00.008501 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:00.008419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" event={"ID":"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e","Type":"ContainerStarted","Data":"e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"}
Jul 29 23:21:00.008501 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:00.008468    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" event={"ID":"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e","Type":"ContainerStarted","Data":"d31877688dc6a731ab84c1354f4e290b913a5b727077988162df34b5f8f7cd01"}
Jul 29 23:21:00.025623 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:00.025567    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" podStartSLOduration=1.025552474 podStartE2EDuration="1.025552474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:21:00.024413249 +0000 UTC m=+2075.933103608" watchObservedRunningTime="2026-07-29 23:21:00.025552474 +0000 UTC m=+2075.934242830"
Jul 29 23:21:00.597925 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:00.597895    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:21:09.598274 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:09.598241    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:21:09.610758 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:09.610733    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:21:10.074757 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:10.074727    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:21:17.690777 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:17.690741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c6vx2"
Jul 29 23:21:33.165677 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.165641    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"]
Jul 29 23:21:33.175993 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:21:33.175942    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:21:33.175993 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:21:33.175993 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:21:33.175993 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18" cmd=["/bin/sleep","15"]
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.176019    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" message="" err=<
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:21:33.176038    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" podUID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" containerName="main" containerID="cri-o://e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"
Jul 29 23:21:33.176348 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.176110    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" podUID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" containerName="main" containerID="cri-o://e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18" gracePeriod=60
Jul 29 23:21:33.537227 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.537203    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:21:33.580611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.580581    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-dshm\") pod \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") "
Jul 29 23:21:33.580836 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.580649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hslfh\" (UniqueName: \"kubernetes.io/projected/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-kube-api-access-hslfh\") pod \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") "
Jul 29 23:21:33.580836 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.580670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tls-certs\") pod \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") "
Jul 29 23:21:33.580836 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.580700    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-home\") pod \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") "
Jul 29 23:21:33.580836 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.580729    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-model-cache\") pod \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") "
Jul 29 23:21:33.580836 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.580743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tmp-dir\") pod \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\" (UID: \"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e\") "
Jul 29 23:21:33.581136 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.581104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-home" (OuterVolumeSpecName: "home") pod "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" (UID: "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:21:33.581191 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.581102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" (UID: "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:21:33.581346 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.581324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" (UID: "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:21:33.582914 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.582888    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-dshm" (OuterVolumeSpecName: "dshm") pod "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" (UID: "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:21:33.583031 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.583012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-kube-api-access-hslfh" (OuterVolumeSpecName: "kube-api-access-hslfh") pod "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" (UID: "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e"). InnerVolumeSpecName "kube-api-access-hslfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:21:33.583308 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.583291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" (UID: "18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:21:33.682090 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.682052    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:21:33.682090 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.682082    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:21:33.682090 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.682091    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:21:33.682090 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.682099    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:21:33.682373 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.682106    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:21:33.682373 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:33.682114    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hslfh\" (UniqueName: \"kubernetes.io/projected/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e-kube-api-access-hslfh\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:21:34.205477 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.205439    2580 generic.go:356] "Generic (PLEG): container finished" podID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" containerID="e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18" exitCode=0
Jul 29 23:21:34.205898 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.205496    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"
Jul 29 23:21:34.205898 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.205526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" event={"ID":"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e","Type":"ContainerDied","Data":"e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"}
Jul 29 23:21:34.205898 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.205587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf" event={"ID":"18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e","Type":"ContainerDied","Data":"d31877688dc6a731ab84c1354f4e290b913a5b727077988162df34b5f8f7cd01"}
Jul 29 23:21:34.205898 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.205610    2580 scope.go:117] "RemoveContainer" containerID="e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"
Jul 29 23:21:34.215692 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.215674    2580 scope.go:117] "RemoveContainer" containerID="e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"
Jul 29 23:21:34.215933 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:21:34.215916    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18\": container with ID starting with e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18 not found: ID does not exist" containerID="e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"
Jul 29 23:21:34.215989 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.215949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18"} err="failed to get container status \"e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18\": rpc error: code = NotFound desc = could not find container \"e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18\": container with ID starting with e17bffbe44199cca1dc7ad792c0067512a7ea21d40ba7ecfe31a9e0f9951aa18 not found: ID does not exist"
Jul 29 23:21:34.226154 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.226127    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"]
Jul 29 23:21:34.228551 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.228528    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-qjflf"]
Jul 29 23:21:34.569506 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:34.569474    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" path="/var/lib/kubelet/pods/18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e/volumes"
Jul 29 23:21:59.495974 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.495942    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"]
Jul 29 23:21:59.496489 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.496474    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" containerName="main"
Jul 29 23:21:59.496562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.496493    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" containerName="main"
Jul 29 23:21:59.496630 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.496614    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b50d2b-c8ee-4672-9e45-4a4d9d64ae3e" containerName="main"
Jul 29 23:21:59.499517 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.499499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.502562 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.502536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 29 23:21:59.502689 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.502560    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-56mjh\""
Jul 29 23:21:59.502689 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.502584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 29 23:21:59.502689 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.502543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 29 23:21:59.507624 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.507603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"]
Jul 29 23:21:59.613293 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.613260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.613468 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.613323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.613468 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.613394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kj2l\" (UniqueName: \"kubernetes.io/projected/c14260f7-3ea2-4186-9e5b-6df4b4008110-kube-api-access-6kj2l\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.613468 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.613442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14260f7-3ea2-4186-9e5b-6df4b4008110-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.613594 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.613493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.613594 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.613514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714236 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kj2l\" (UniqueName: \"kubernetes.io/projected/c14260f7-3ea2-4186-9e5b-6df4b4008110-kube-api-access-6kj2l\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14260f7-3ea2-4186-9e5b-6df4b4008110-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714592 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714630 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714695 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714742 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.714933 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.714903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.715073 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.715052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.716741 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.716719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.716888 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.716872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14260f7-3ea2-4186-9e5b-6df4b4008110-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.725585 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.725550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kj2l\" (UniqueName: \"kubernetes.io/projected/c14260f7-3ea2-4186-9e5b-6df4b4008110-kube-api-access-6kj2l\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.811277 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.811247    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:21:59.946689 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:21:59.946663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"]
Jul 29 23:21:59.948633 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:21:59.948595    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14260f7_3ea2_4186_9e5b_6df4b4008110.slice/crio-83ae488c6313c75047c82d5593022f2fc1877bc95723f6854cc6bfe571f07a2a WatchSource:0}: Error finding container 83ae488c6313c75047c82d5593022f2fc1877bc95723f6854cc6bfe571f07a2a: Status 404 returned error can't find the container with id 83ae488c6313c75047c82d5593022f2fc1877bc95723f6854cc6bfe571f07a2a
Jul 29 23:22:00.347893 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:00.347856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" event={"ID":"c14260f7-3ea2-4186-9e5b-6df4b4008110","Type":"ContainerStarted","Data":"a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"}
Jul 29 23:22:00.348085 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:00.347914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" event={"ID":"c14260f7-3ea2-4186-9e5b-6df4b4008110","Type":"ContainerStarted","Data":"83ae488c6313c75047c82d5593022f2fc1877bc95723f6854cc6bfe571f07a2a"}
Jul 29 23:22:00.367724 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:00.367658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" podStartSLOduration=1.367642553 podStartE2EDuration="1.367642553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:22:00.366415681 +0000 UTC m=+2136.275106041" watchObservedRunningTime="2026-07-29 23:22:00.367642553 +0000 UTC m=+2136.276332911"
Jul 29 23:22:00.811867 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:00.811837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:22:09.811409 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:09.811366    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:22:09.824443 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:09.824412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:22:10.416104 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:22:10.416069    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:23:35.059512 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.059474    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"]
Jul 29 23:23:35.068585 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:23:35.068548    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:23:35.068585 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:23:35.068585 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:23:35.068585 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490" cmd=["/bin/sleep","15"]
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.068601    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" message="" err=<
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:23:35.068614    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerName="main" containerID="cri-o://a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"
Jul 29 23:23:35.068912 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.068649    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerName="main" containerID="cri-o://a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490" gracePeriod=60
Jul 29 23:23:35.405219 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.405123    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 29 23:23:35.821495 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.821467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:23:35.873570 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.873541    2580 generic.go:356] "Generic (PLEG): container finished" podID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerID="a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490" exitCode=0
Jul 29 23:23:35.873751 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.873627    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"
Jul 29 23:23:35.873751 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.873646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" event={"ID":"c14260f7-3ea2-4186-9e5b-6df4b4008110","Type":"ContainerDied","Data":"a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"}
Jul 29 23:23:35.873855 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.873752    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr" event={"ID":"c14260f7-3ea2-4186-9e5b-6df4b4008110","Type":"ContainerDied","Data":"83ae488c6313c75047c82d5593022f2fc1877bc95723f6854cc6bfe571f07a2a"}
Jul 29 23:23:35.873855 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.873797    2580 scope.go:117] "RemoveContainer" containerID="a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"
Jul 29 23:23:35.876756 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.876734    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-tmp-dir\") pod \"c14260f7-3ea2-4186-9e5b-6df4b4008110\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") "
Jul 29 23:23:35.876883 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.876824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kj2l\" (UniqueName: \"kubernetes.io/projected/c14260f7-3ea2-4186-9e5b-6df4b4008110-kube-api-access-6kj2l\") pod \"c14260f7-3ea2-4186-9e5b-6df4b4008110\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") "
Jul 29 23:23:35.876883 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.876855    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-home\") pod \"c14260f7-3ea2-4186-9e5b-6df4b4008110\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") "
Jul 29 23:23:35.876983 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.876908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14260f7-3ea2-4186-9e5b-6df4b4008110-tls-certs\") pod \"c14260f7-3ea2-4186-9e5b-6df4b4008110\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") "
Jul 29 23:23:35.876983 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.876944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-dshm\") pod \"c14260f7-3ea2-4186-9e5b-6df4b4008110\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") "
Jul 29 23:23:35.877089 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.877012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-model-cache\") pod \"c14260f7-3ea2-4186-9e5b-6df4b4008110\" (UID: \"c14260f7-3ea2-4186-9e5b-6df4b4008110\") "
Jul 29 23:23:35.877141 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.877099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c14260f7-3ea2-4186-9e5b-6df4b4008110" (UID: "c14260f7-3ea2-4186-9e5b-6df4b4008110"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:35.877227 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.877202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-home" (OuterVolumeSpecName: "home") pod "c14260f7-3ea2-4186-9e5b-6df4b4008110" (UID: "c14260f7-3ea2-4186-9e5b-6df4b4008110"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:35.877354 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.877332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-model-cache" (OuterVolumeSpecName: "model-cache") pod "c14260f7-3ea2-4186-9e5b-6df4b4008110" (UID: "c14260f7-3ea2-4186-9e5b-6df4b4008110"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:35.877469 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.877367    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:23:35.877469 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.877407    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:23:35.879331 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.879303    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-dshm" (OuterVolumeSpecName: "dshm") pod "c14260f7-3ea2-4186-9e5b-6df4b4008110" (UID: "c14260f7-3ea2-4186-9e5b-6df4b4008110"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:23:35.879331 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.879316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14260f7-3ea2-4186-9e5b-6df4b4008110-kube-api-access-6kj2l" (OuterVolumeSpecName: "kube-api-access-6kj2l") pod "c14260f7-3ea2-4186-9e5b-6df4b4008110" (UID: "c14260f7-3ea2-4186-9e5b-6df4b4008110"). InnerVolumeSpecName "kube-api-access-6kj2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:23:35.879480 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.879417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14260f7-3ea2-4186-9e5b-6df4b4008110-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c14260f7-3ea2-4186-9e5b-6df4b4008110" (UID: "c14260f7-3ea2-4186-9e5b-6df4b4008110"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:23:35.883490 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.883463    2580 scope.go:117] "RemoveContainer" containerID="a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"
Jul 29 23:23:35.883860 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:23:35.883834    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490\": container with ID starting with a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490 not found: ID does not exist" containerID="a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"
Jul 29 23:23:35.883951 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.883871    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490"} err="failed to get container status \"a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490\": rpc error: code = NotFound desc = could not find container \"a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490\": container with ID starting with a06423dbb49e65278c4f43d9fd5f3df3630fe27f6f4a3683e5fc242f19406490 not found: ID does not exist"
Jul 29 23:23:35.978740 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.978660    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kj2l\" (UniqueName: \"kubernetes.io/projected/c14260f7-3ea2-4186-9e5b-6df4b4008110-kube-api-access-6kj2l\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:23:35.978740 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.978686    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c14260f7-3ea2-4186-9e5b-6df4b4008110-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:23:35.978740 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.978697    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:23:35.978740 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:35.978704    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c14260f7-3ea2-4186-9e5b-6df4b4008110-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:23:36.199877 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:36.199846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"]
Jul 29 23:23:36.203516 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:36.203492    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-v48pr"]
Jul 29 23:23:36.570571 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:23:36.570538    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" path="/var/lib/kubelet/pods/c14260f7-3ea2-4186-9e5b-6df4b4008110/volumes"
Jul 29 23:24:03.674100 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.674068    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"]
Jul 29 23:24:03.674496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.674436    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerName="main"
Jul 29 23:24:03.674496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.674448    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerName="main"
Jul 29 23:24:03.674575 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.674518    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14260f7-3ea2-4186-9e5b-6df4b4008110" containerName="main"
Jul 29 23:24:03.677645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.677615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.680407 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.680377    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 29 23:24:03.680549 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.680441    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 29 23:24:03.680549 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.680509    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 29 23:24:03.680666 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.680552    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-x9ppl\""
Jul 29 23:24:03.685996 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.685971    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"]
Jul 29 23:24:03.818882 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.818831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.818882 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.818887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.819113 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.818909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.819113 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.818924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.819113 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.818972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.819113 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.818988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hb4\" (UniqueName: \"kubernetes.io/projected/3c3585de-c8da-4276-b6ae-7b433e9d39eb-kube-api-access-g6hb4\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920270 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920270 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hb4\" (UniqueName: \"kubernetes.io/projected/3c3585de-c8da-4276-b6ae-7b433e9d39eb-kube-api-access-g6hb4\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920701 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920749 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.920801 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.920761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.922634 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.922612    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.922991 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.922941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.931012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.930936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hb4\" (UniqueName: \"kubernetes.io/projected/3c3585de-c8da-4276-b6ae-7b433e9d39eb-kube-api-access-g6hb4\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:03.989706 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:03.989668    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:04.127679 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:04.127651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"]
Jul 29 23:24:04.130167 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:24:04.130139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3585de_c8da_4276_b6ae_7b433e9d39eb.slice/crio-db756d8712ed18a3a9b95caf7f10eab31ef8414faeceef019d41f87302e96eb6 WatchSource:0}: Error finding container db756d8712ed18a3a9b95caf7f10eab31ef8414faeceef019d41f87302e96eb6: Status 404 returned error can't find the container with id db756d8712ed18a3a9b95caf7f10eab31ef8414faeceef019d41f87302e96eb6
Jul 29 23:24:05.030297 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:05.030265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" event={"ID":"3c3585de-c8da-4276-b6ae-7b433e9d39eb","Type":"ContainerStarted","Data":"4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"}
Jul 29 23:24:05.030781 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:05.030320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" event={"ID":"3c3585de-c8da-4276-b6ae-7b433e9d39eb","Type":"ContainerStarted","Data":"db756d8712ed18a3a9b95caf7f10eab31ef8414faeceef019d41f87302e96eb6"}
Jul 29 23:24:05.049738 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:05.049682    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" podStartSLOduration=2.04966742 podStartE2EDuration="2.04966742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:24:05.047385595 +0000 UTC m=+2260.956075955" watchObservedRunningTime="2026-07-29 23:24:05.04966742 +0000 UTC m=+2260.958357779"
Jul 29 23:24:05.990684 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:05.990650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:13.990228 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:13.990186    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:14.003012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:14.002984    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:24:14.087219 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:24:14.087178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:25:42.360892 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.360781    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"]
Jul 29 23:25:42.370349 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:25:42.370311    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 23:25:42.370349 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:25:42.370349 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:25:42.370349 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf" cmd=["/bin/sleep","15"]
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.370373    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" message="" err=<
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:25:42.370386    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" podUID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" containerName="main" containerID="cri-o://4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"
Jul 29 23:25:42.370611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.370433    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" podUID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" containerName="main" containerID="cri-o://4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf" gracePeriod=60
Jul 29 23:25:42.737768 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.737737    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:25:42.872373 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-home\") pod \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") "
Jul 29 23:25:42.872590 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872419    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tls-certs\") pod \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") "
Jul 29 23:25:42.872590 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-model-cache\") pod \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") "
Jul 29 23:25:42.872590 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872486    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6hb4\" (UniqueName: \"kubernetes.io/projected/3c3585de-c8da-4276-b6ae-7b433e9d39eb-kube-api-access-g6hb4\") pod \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") "
Jul 29 23:25:42.872590 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872559    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tmp-dir\") pod \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") "
Jul 29 23:25:42.872590 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-dshm\") pod \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\" (UID: \"3c3585de-c8da-4276-b6ae-7b433e9d39eb\") "
Jul 29 23:25:42.872923 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872641    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-home" (OuterVolumeSpecName: "home") pod "3c3585de-c8da-4276-b6ae-7b433e9d39eb" (UID: "3c3585de-c8da-4276-b6ae-7b433e9d39eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:42.872923 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c3585de-c8da-4276-b6ae-7b433e9d39eb" (UID: "3c3585de-c8da-4276-b6ae-7b433e9d39eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:42.872923 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c3585de-c8da-4276-b6ae-7b433e9d39eb" (UID: "3c3585de-c8da-4276-b6ae-7b433e9d39eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:42.872923 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872919    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:25:42.873128 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872938    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:25:42.873128 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.872952    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:25:42.875135 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.875106    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-dshm" (OuterVolumeSpecName: "dshm") pod "3c3585de-c8da-4276-b6ae-7b433e9d39eb" (UID: "3c3585de-c8da-4276-b6ae-7b433e9d39eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:25:42.875135 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.875108    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c3585de-c8da-4276-b6ae-7b433e9d39eb" (UID: "3c3585de-c8da-4276-b6ae-7b433e9d39eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:25:42.875306 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.875153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3585de-c8da-4276-b6ae-7b433e9d39eb-kube-api-access-g6hb4" (OuterVolumeSpecName: "kube-api-access-g6hb4") pod "3c3585de-c8da-4276-b6ae-7b433e9d39eb" (UID: "3c3585de-c8da-4276-b6ae-7b433e9d39eb"). InnerVolumeSpecName "kube-api-access-g6hb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:25:42.974051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.973964    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hb4\" (UniqueName: \"kubernetes.io/projected/3c3585de-c8da-4276-b6ae-7b433e9d39eb-kube-api-access-g6hb4\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:25:42.974051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.973992    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3585de-c8da-4276-b6ae-7b433e9d39eb-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:25:42.974051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:42.974001    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3585de-c8da-4276-b6ae-7b433e9d39eb-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:25:43.558363 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.558328    2580 generic.go:356] "Generic (PLEG): container finished" podID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" containerID="4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf" exitCode=0
Jul 29 23:25:43.558878 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.558385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"
Jul 29 23:25:43.558878 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.558406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" event={"ID":"3c3585de-c8da-4276-b6ae-7b433e9d39eb","Type":"ContainerDied","Data":"4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"}
Jul 29 23:25:43.558878 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.558459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm" event={"ID":"3c3585de-c8da-4276-b6ae-7b433e9d39eb","Type":"ContainerDied","Data":"db756d8712ed18a3a9b95caf7f10eab31ef8414faeceef019d41f87302e96eb6"}
Jul 29 23:25:43.558878 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.558481    2580 scope.go:117] "RemoveContainer" containerID="4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"
Jul 29 23:25:43.568608 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.568589    2580 scope.go:117] "RemoveContainer" containerID="4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"
Jul 29 23:25:43.568896 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:25:43.568878    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf\": container with ID starting with 4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf not found: ID does not exist" containerID="4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"
Jul 29 23:25:43.568948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.568906    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf"} err="failed to get container status \"4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf\": rpc error: code = NotFound desc = could not find container \"4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf\": container with ID starting with 4bb1ca4c4a796e8313eef0ba65e94c8e08b2aad503e287a8d5de9254629350cf not found: ID does not exist"
Jul 29 23:25:43.579388 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.579365    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"]
Jul 29 23:25:43.585084 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:43.585062    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969sbghm"]
Jul 29 23:25:44.569366 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:25:44.569332    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" path="/var/lib/kubelet/pods/3c3585de-c8da-4276-b6ae-7b433e9d39eb/volumes"
Jul 29 23:26:09.108842 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.108779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"]
Jul 29 23:26:09.109355 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.109339    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" containerName="main"
Jul 29 23:26:09.109402 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.109359    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" containerName="main"
Jul 29 23:26:09.109477 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.109466    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3585de-c8da-4276-b6ae-7b433e9d39eb" containerName="main"
Jul 29 23:26:09.112912 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.112891    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.115237 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.115036    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 29 23:26:09.115237 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.115084    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 29 23:26:09.115237 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.115128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 29 23:26:09.115237 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.115134    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 29 23:26:09.115237 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.115138    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-7r2pc\""
Jul 29 23:26:09.115662 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.115313    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 23:26:09.123992 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.123968    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"]
Jul 29 23:26:09.201900 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.201862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202091 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.201913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202091 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.201991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202091 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.202051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202207 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.202105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcs4\" (UniqueName: \"kubernetes.io/projected/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kube-api-access-cbcs4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202207 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.202137    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202207 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.202177    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.202207 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.202198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.302770 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcs4\" (UniqueName: \"kubernetes.io/projected/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kube-api-access-cbcs4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303012 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.302999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303403 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.303302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303403 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.303324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303516 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.303425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303516 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.303456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.303665 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.303645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.305190 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.305170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.305603 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.305582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.312068 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.312042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcs4\" (UniqueName: \"kubernetes.io/projected/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kube-api-access-cbcs4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.425601 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.425502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:09.576129 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.576103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"]
Jul 29 23:26:09.578156 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:26:09.578121    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998db1ad_8c0a_45fb_8e1d_c30ecc7122f0.slice/crio-d3062b21e3ac55fba9424ecf7f9bb5ca864c1e84492a006abe3628292baf0184 WatchSource:0}: Error finding container d3062b21e3ac55fba9424ecf7f9bb5ca864c1e84492a006abe3628292baf0184: Status 404 returned error can't find the container with id d3062b21e3ac55fba9424ecf7f9bb5ca864c1e84492a006abe3628292baf0184
Jul 29 23:26:09.696046 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.695957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" event={"ID":"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0","Type":"ContainerStarted","Data":"2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811"}
Jul 29 23:26:09.696046 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:09.696026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" event={"ID":"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0","Type":"ContainerStarted","Data":"d3062b21e3ac55fba9424ecf7f9bb5ca864c1e84492a006abe3628292baf0184"}
Jul 29 23:26:16.737061 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:16.737025    2580 generic.go:356] "Generic (PLEG): container finished" podID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerID="2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811" exitCode=0
Jul 29 23:26:16.737627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:16.737078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" event={"ID":"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0","Type":"ContainerDied","Data":"2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811"}
Jul 29 23:26:17.744950 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:17.744916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" event={"ID":"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0","Type":"ContainerStarted","Data":"59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d"}
Jul 29 23:26:17.767131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:17.767074    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podStartSLOduration=2.162402153 podStartE2EDuration="8.767057577s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.604655424s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:26:17.764609594 +0000 UTC m=+2393.673299952" watchObservedRunningTime="2026-07-29 23:26:17.767057577 +0000 UTC m=+2393.675747935"
Jul 29 23:26:18.426443 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:18.426405    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:19.426525 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:19.426479    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:26:19.428649 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:19.428616    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:26:29.426945 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:29.426902    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:26:39.426314 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:39.426271    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:26:49.426409 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:49.426282    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:26:59.426251 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:26:59.426212    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:27:09.426248 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:27:09.426198    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:27:19.426172 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:27:19.426124    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:27:29.425923 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:27:29.425880    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:27:39.436680 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:27:39.436651    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:27:39.444441 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:27:39.444417    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:28:04.355057 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:04.355019    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"]
Jul 29 23:28:19.378693 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:19.378598    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" containerID="cri-o://59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d" gracePeriod=45
Jul 29 23:28:19.438302 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:19.438260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 29 23:28:20.040574 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.040549    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:28:20.101529 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101446    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kserve-provision-location\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101529 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101481    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbcs4\" (UniqueName: \"kubernetes.io/projected/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kube-api-access-cbcs4\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101529 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-cabundle-cert\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101850 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-dshm\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101850 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101558    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tmp-dir\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101850 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-home\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101850 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tls-certs\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.101850 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101678    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-model-cache\") pod \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\" (UID: \"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0\") "
Jul 29 23:28:20.102132 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.101973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 23:28:20.102132 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.102115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:28:20.102241 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.102147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-home" (OuterVolumeSpecName: "home") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:28:20.103778 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.103749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-dshm" (OuterVolumeSpecName: "dshm") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:28:20.103974 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.103956    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kube-api-access-cbcs4" (OuterVolumeSpecName: "kube-api-access-cbcs4") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "kube-api-access-cbcs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:28:20.104403 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.104380    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:28:20.119059 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.119023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:28:20.169627 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.169583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" (UID: "998db1ad-8c0a-45fb-8e1d-c30ecc7122f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:28:20.202256 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202226    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202256 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202257    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcs4\" (UniqueName: \"kubernetes.io/projected/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-kube-api-access-cbcs4\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202269    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-cabundle-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202278    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202287    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202294    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202301    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.202410 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.202308    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:28:20.481639 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.481545    2580 generic.go:356] "Generic (PLEG): container finished" podID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerID="59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d" exitCode=0
Jul 29 23:28:20.481639 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.481604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"
Jul 29 23:28:20.481639 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.481611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" event={"ID":"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0","Type":"ContainerDied","Data":"59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d"}
Jul 29 23:28:20.482309 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.481656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb" event={"ID":"998db1ad-8c0a-45fb-8e1d-c30ecc7122f0","Type":"ContainerDied","Data":"d3062b21e3ac55fba9424ecf7f9bb5ca864c1e84492a006abe3628292baf0184"}
Jul 29 23:28:20.482309 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.481678    2580 scope.go:117] "RemoveContainer" containerID="59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d"
Jul 29 23:28:20.499971 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.499949    2580 scope.go:117] "RemoveContainer" containerID="2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811"
Jul 29 23:28:20.508838 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.508808    2580 scope.go:117] "RemoveContainer" containerID="59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d"
Jul 29 23:28:20.509210 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:28:20.509185    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d\": container with ID starting with 59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d not found: ID does not exist" containerID="59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d"
Jul 29 23:28:20.509298 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.509216    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d"} err="failed to get container status \"59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d\": rpc error: code = NotFound desc = could not find container \"59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d\": container with ID starting with 59b6caddec5af6f9ce76c3aae271da5e081396ed1d8d461fa30aca54c8a3929d not found: ID does not exist"
Jul 29 23:28:20.509298 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.509235    2580 scope.go:117] "RemoveContainer" containerID="2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811"
Jul 29 23:28:20.509475 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:28:20.509456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811\": container with ID starting with 2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811 not found: ID does not exist" containerID="2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811"
Jul 29 23:28:20.509513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.509481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811"} err="failed to get container status \"2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811\": rpc error: code = NotFound desc = could not find container \"2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811\": container with ID starting with 2ea147128e9b2fdd26d6b33193902fabc326aab98e967876afa6b71c9e091811 not found: ID does not exist"
Jul 29 23:28:20.509513 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.509504    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"]
Jul 29 23:28:20.515438 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.515415    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-576b9f94cb-lqjhb"]
Jul 29 23:28:20.570206 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:20.570175    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" path="/var/lib/kubelet/pods/998db1ad-8c0a-45fb-8e1d-c30ecc7122f0/volumes"
Jul 29 23:28:28.183538 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.183509    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"]
Jul 29 23:28:28.183948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.183892    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="storage-initializer"
Jul 29 23:28:28.183948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.183904    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="storage-initializer"
Jul 29 23:28:28.183948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.183930    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main"
Jul 29 23:28:28.183948 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.183936    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main"
Jul 29 23:28:28.184084 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.183987    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="998db1ad-8c0a-45fb-8e1d-c30ecc7122f0" containerName="main"
Jul 29 23:28:28.188408 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.188391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.190717 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.190685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-vf9bf\""
Jul 29 23:28:28.190890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.190738    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 29 23:28:28.190890 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.190777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 23:28:28.191032 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.191017    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 29 23:28:28.191354 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.191335    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 29 23:28:28.191455 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.191368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 29 23:28:28.200335 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.200315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"]
Jul 29 23:28:28.259650 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c37d75-ef75-4628-8cc4-b4298aea8711-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.259777 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.259777 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ps7\" (UniqueName: \"kubernetes.io/projected/30c37d75-ef75-4628-8cc4-b4298aea8711-kube-api-access-s7ps7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.259777 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.259913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c37d75-ef75-4628-8cc4-b4298aea8711-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.259913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.259913 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.260040 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.259923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c37d75-ef75-4628-8cc4-b4298aea8711-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360773 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360822    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c37d75-ef75-4628-8cc4-b4298aea8711-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ps7\" (UniqueName: \"kubernetes.io/projected/30c37d75-ef75-4628-8cc4-b4298aea8711-kube-api-access-s7ps7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.360988 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.360984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.361311 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.361277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.361394 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.361364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.361466 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.361443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.361517 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.361498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.361648 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.361633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c37d75-ef75-4628-8cc4-b4298aea8711-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.363144 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.363121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.363372 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.363354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c37d75-ef75-4628-8cc4-b4298aea8711-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.368542 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.368523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ps7\" (UniqueName: \"kubernetes.io/projected/30c37d75-ef75-4628-8cc4-b4298aea8711-kube-api-access-s7ps7\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.499642 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.499557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:28.643359 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:28.643332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"]
Jul 29 23:28:28.645342 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:28:28.645314    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c37d75_ef75_4628_8cc4_b4298aea8711.slice/crio-451a6e67f9e397557efebc08dbf89bedbf298b69b114a808c867c7cce0d00bc6 WatchSource:0}: Error finding container 451a6e67f9e397557efebc08dbf89bedbf298b69b114a808c867c7cce0d00bc6: Status 404 returned error can't find the container with id 451a6e67f9e397557efebc08dbf89bedbf298b69b114a808c867c7cce0d00bc6
Jul 29 23:28:29.532015 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:29.531979    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" event={"ID":"30c37d75-ef75-4628-8cc4-b4298aea8711","Type":"ContainerStarted","Data":"abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6"}
Jul 29 23:28:29.532422 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:29.532033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" event={"ID":"30c37d75-ef75-4628-8cc4-b4298aea8711","Type":"ContainerStarted","Data":"451a6e67f9e397557efebc08dbf89bedbf298b69b114a808c867c7cce0d00bc6"}
Jul 29 23:28:34.563813 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:34.563764    2580 generic.go:356] "Generic (PLEG): container finished" podID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerID="abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6" exitCode=0
Jul 29 23:28:34.569604 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:34.569576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" event={"ID":"30c37d75-ef75-4628-8cc4-b4298aea8711","Type":"ContainerDied","Data":"abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6"}
Jul 29 23:28:35.572083 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:35.572037    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" event={"ID":"30c37d75-ef75-4628-8cc4-b4298aea8711","Type":"ContainerStarted","Data":"7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8"}
Jul 29 23:28:35.596143 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:35.596080    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podStartSLOduration=2.626044832 podStartE2EDuration="7.596057801s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.970012969s" isStatefulPod=false podCreationTimestamp="2026-07-29 23:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:28:35.594047836 +0000 UTC m=+2531.502738194" watchObservedRunningTime="2026-07-29 23:28:35.596057801 +0000 UTC m=+2531.504748160"
Jul 29 23:28:36.500052 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:36.500009    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:38.500435 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:38.500376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:28:38.501805 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:38.501759    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:28:48.500874 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:48.500813    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:28:58.500216 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:28:58.500177    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:29:08.501028 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:08.500976    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:29:18.500107 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:18.500057    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:29:28.500660 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:28.500619    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:29:38.500329 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:38.500272    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:29:48.500549 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:48.500452    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:29:58.516164 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:58.516135    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:29:58.528351 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:29:58.528318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:30:00.167108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:00.167075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"]
Jul 29 23:30:15.190570 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.190511    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" containerID="cri-o://7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8" gracePeriod=45
Jul 29 23:30:15.517114 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.517011    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 29 23:30:15.746945 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.746921    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:30:15.844985 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.844948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-model-cache\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845177 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.844993    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-home\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845177 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-kserve-provision-location\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845177 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c37d75-ef75-4628-8cc4-b4298aea8711-cabundle-cert\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845177 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845081    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c37d75-ef75-4628-8cc4-b4298aea8711-tls-certs\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845177 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845126    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-dshm\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845457 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845289    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-tmp-dir\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845457 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-model-cache" (OuterVolumeSpecName: "model-cache") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:30:15.845457 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7ps7\" (UniqueName: \"kubernetes.io/projected/30c37d75-ef75-4628-8cc4-b4298aea8711-kube-api-access-s7ps7\") pod \"30c37d75-ef75-4628-8cc4-b4298aea8711\" (UID: \"30c37d75-ef75-4628-8cc4-b4298aea8711\") "
Jul 29 23:30:15.845457 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-home" (OuterVolumeSpecName: "home") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:30:15.845675 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c37d75-ef75-4628-8cc4-b4298aea8711-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 23:30:15.845774 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:30:15.845774 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845766    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.845924 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845815    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.845924 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.845830    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30c37d75-ef75-4628-8cc4-b4298aea8711-cabundle-cert\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.847421 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.847387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c37d75-ef75-4628-8cc4-b4298aea8711-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:30:15.847421 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.847404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-dshm" (OuterVolumeSpecName: "dshm") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:30:15.847588 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.847565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c37d75-ef75-4628-8cc4-b4298aea8711-kube-api-access-s7ps7" (OuterVolumeSpecName: "kube-api-access-s7ps7") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "kube-api-access-s7ps7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:30:15.908425 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.908385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30c37d75-ef75-4628-8cc4-b4298aea8711" (UID: "30c37d75-ef75-4628-8cc4-b4298aea8711"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:30:15.946352 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.946310    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c37d75-ef75-4628-8cc4-b4298aea8711-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.946352 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.946339    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.946352 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.946358    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.946599 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.946371    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ps7\" (UniqueName: \"kubernetes.io/projected/30c37d75-ef75-4628-8cc4-b4298aea8711-kube-api-access-s7ps7\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:15.946599 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:15.946384    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30c37d75-ef75-4628-8cc4-b4298aea8711-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:16.185994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.185896    2580 generic.go:356] "Generic (PLEG): container finished" podID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerID="7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8" exitCode=0
Jul 29 23:30:16.185994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.185953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" event={"ID":"30c37d75-ef75-4628-8cc4-b4298aea8711","Type":"ContainerDied","Data":"7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8"}
Jul 29 23:30:16.185994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.185960    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"
Jul 29 23:30:16.185994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.185998    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc" event={"ID":"30c37d75-ef75-4628-8cc4-b4298aea8711","Type":"ContainerDied","Data":"451a6e67f9e397557efebc08dbf89bedbf298b69b114a808c867c7cce0d00bc6"}
Jul 29 23:30:16.186338 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.186020    2580 scope.go:117] "RemoveContainer" containerID="7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8"
Jul 29 23:30:16.195484 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.195279    2580 scope.go:117] "RemoveContainer" containerID="abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6"
Jul 29 23:30:16.203996 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.203978    2580 scope.go:117] "RemoveContainer" containerID="7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8"
Jul 29 23:30:16.204284 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:30:16.204239    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8\": container with ID starting with 7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8 not found: ID does not exist" containerID="7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8"
Jul 29 23:30:16.204393 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.204283    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8"} err="failed to get container status \"7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8\": rpc error: code = NotFound desc = could not find container \"7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8\": container with ID starting with 7cb215a0bbec918b4d5f2c14a566ab5d3c94f50ffa8a9279472d438ce59deda8 not found: ID does not exist"
Jul 29 23:30:16.204393 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.204307    2580 scope.go:117] "RemoveContainer" containerID="abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6"
Jul 29 23:30:16.205138 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:30:16.205112    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6\": container with ID starting with abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6 not found: ID does not exist" containerID="abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6"
Jul 29 23:30:16.205219 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.205147    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6"} err="failed to get container status \"abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6\": rpc error: code = NotFound desc = could not find container \"abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6\": container with ID starting with abaf6c06d8d4cb756fc26bf2b097d49ae2abb44d08162d62b739f679fa06bdd6 not found: ID does not exist"
Jul 29 23:30:16.207294 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.207274    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"]
Jul 29 23:30:16.211176 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.211155    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78c7cc8d9-xjjsc"]
Jul 29 23:30:16.569877 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:16.569848    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" path="/var/lib/kubelet/pods/30c37d75-ef75-4628-8cc4-b4298aea8711/volumes"
Jul 29 23:30:23.409138 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.409096    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"]
Jul 29 23:30:23.409656 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.409632    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main"
Jul 29 23:30:23.409656 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.409651    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main"
Jul 29 23:30:23.409774 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.409689    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="storage-initializer"
Jul 29 23:30:23.409774 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.409700    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="storage-initializer"
Jul 29 23:30:23.409912 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.409804    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c37d75-ef75-4628-8cc4-b4298aea8711" containerName="main"
Jul 29 23:30:23.413331 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.413310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:23.415493 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.415471    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 29 23:30:23.415602 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.415494    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 29 23:30:23.416132 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.416105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-7txlv\""
Jul 29 23:30:23.418747 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.418728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"]
Jul 29 23:30:23.509434 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.509400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ww2\" (UniqueName: \"kubernetes.io/projected/8af150cd-149b-4dbb-a901-f991ea198969-kube-api-access-t6ww2\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:23.610131 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.610094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ww2\" (UniqueName: \"kubernetes.io/projected/8af150cd-149b-4dbb-a901-f991ea198969-kube-api-access-t6ww2\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:23.617745 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:23.617721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ww2\" (UniqueName: \"kubernetes.io/projected/8af150cd-149b-4dbb-a901-f991ea198969-kube-api-access-t6ww2\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:27.844809 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:27.844746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:27.849482 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:27.849460    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:27.946246 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:27.946208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:27.946415 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:27.946315    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:27.948738 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:27.948720    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") DevicePath \"csi-54b13d7209a401aa8c0f2432aea2d571fbb1056184ad7cd1c9a32ca59ae4b7be\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:28.171862 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:28.171835    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ab6e994f0711b85347795a4cc0207f4d9c0e47d630289dc945bbd82e3298c27e/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:28.188305 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:28.188279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"e2e-pvc-model-download-6kdhv\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:28.225359 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:28.225331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-7txlv\""
Jul 29 23:30:28.233728 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:28.233709    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:28.361960 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:28.361936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"]
Jul 29 23:30:28.364208 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:30:28.364178    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8af150cd_149b_4dbb_a901_f991ea198969.slice/crio-c5fe393a3f418868917d64b0907a60f4fe411993c7089629190e1e77fae5d63c WatchSource:0}: Error finding container c5fe393a3f418868917d64b0907a60f4fe411993c7089629190e1e77fae5d63c: Status 404 returned error can't find the container with id c5fe393a3f418868917d64b0907a60f4fe411993c7089629190e1e77fae5d63c
Jul 29 23:30:29.257259 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:29.257219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv" event={"ID":"8af150cd-149b-4dbb-a901-f991ea198969","Type":"ContainerStarted","Data":"79ba4949a3b880b74330c9547bf7a2034ed280b3e680076424a99bcbc528a1f7"}
Jul 29 23:30:29.257703 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:29.257272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv" event={"ID":"8af150cd-149b-4dbb-a901-f991ea198969","Type":"ContainerStarted","Data":"c5fe393a3f418868917d64b0907a60f4fe411993c7089629190e1e77fae5d63c"}
Jul 29 23:30:29.273078 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:29.273018    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv" podStartSLOduration=9.273001633 podStartE2EDuration="9.273001633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 23:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:30:29.271024531 +0000 UTC m=+2645.179714890" watchObservedRunningTime="2026-07-29 23:30:29.273001633 +0000 UTC m=+2645.181691990"
Jul 29 23:30:35.296504 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:35.296471    2580 generic.go:356] "Generic (PLEG): container finished" podID="8af150cd-149b-4dbb-a901-f991ea198969" containerID="79ba4949a3b880b74330c9547bf7a2034ed280b3e680076424a99bcbc528a1f7" exitCode=0
Jul 29 23:30:35.296954 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:35.296516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv" event={"ID":"8af150cd-149b-4dbb-a901-f991ea198969","Type":"ContainerDied","Data":"79ba4949a3b880b74330c9547bf7a2034ed280b3e680076424a99bcbc528a1f7"}
Jul 29 23:30:36.440372 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.440345    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:36.525950 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.525919    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"8af150cd-149b-4dbb-a901-f991ea198969\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") "
Jul 29 23:30:36.526155 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.526017    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6ww2\" (UniqueName: \"kubernetes.io/projected/8af150cd-149b-4dbb-a901-f991ea198969-kube-api-access-t6ww2\") pod \"8af150cd-149b-4dbb-a901-f991ea198969\" (UID: \"8af150cd-149b-4dbb-a901-f991ea198969\") "
Jul 29 23:30:36.528241 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.528208    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af150cd-149b-4dbb-a901-f991ea198969-kube-api-access-t6ww2" (OuterVolumeSpecName: "kube-api-access-t6ww2") pod "8af150cd-149b-4dbb-a901-f991ea198969" (UID: "8af150cd-149b-4dbb-a901-f991ea198969"). InnerVolumeSpecName "kube-api-access-t6ww2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:30:36.529709 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.529686    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e" (OuterVolumeSpecName: "model-storage") pod "8af150cd-149b-4dbb-a901-f991ea198969" (UID: "8af150cd-149b-4dbb-a901-f991ea198969"). InnerVolumeSpecName "pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 23:30:36.627010 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.626965    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ww2\" (UniqueName: \"kubernetes.io/projected/8af150cd-149b-4dbb-a901-f991ea198969-kube-api-access-t6ww2\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:30:36.627181 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.627029    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") on node \"ip-10-0-131-193.ec2.internal\" "
Jul 29 23:30:36.742674 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.742651    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e") on node "ip-10-0-131-193.ec2.internal"
Jul 29 23:30:36.829146 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:36.829058    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"csi-54b13d7209a401aa8c0f2432aea2d571fbb1056184ad7cd1c9a32ca59ae4b7be\""
Jul 29 23:30:37.308413 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:37.308385    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"
Jul 29 23:30:37.308588 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:37.308427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv" event={"ID":"8af150cd-149b-4dbb-a901-f991ea198969","Type":"ContainerDied","Data":"c5fe393a3f418868917d64b0907a60f4fe411993c7089629190e1e77fae5d63c"}
Jul 29 23:30:37.308588 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:37.308463    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5fe393a3f418868917d64b0907a60f4fe411993c7089629190e1e77fae5d63c"
Jul 29 23:30:52.694134 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.694087    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"]
Jul 29 23:30:52.694722 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.694699    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af150cd-149b-4dbb-a901-f991ea198969" containerName="storage-initializer"
Jul 29 23:30:52.694823 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.694726    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af150cd-149b-4dbb-a901-f991ea198969" containerName="storage-initializer"
Jul 29 23:30:52.694900 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.694866    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af150cd-149b-4dbb-a901-f991ea198969" containerName="storage-initializer"
Jul 29 23:30:52.698007 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.697984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.700234 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.700204    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 29 23:30:52.700362 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.700258    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 29 23:30:52.700552 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.700533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-7txlv\""
Jul 29 23:30:52.700823 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.700807    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 29 23:30:52.704887 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.704861    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"]
Jul 29 23:30:52.772162 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.772124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.772350 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.772187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.772350 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.772216    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.772350 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.772242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7d7\" (UniqueName: \"kubernetes.io/projected/59995b9b-a14b-48fd-9844-0b46eb26cad5-kube-api-access-xh7d7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.772350 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.772317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.772496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.772351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59995b9b-a14b-48fd-9844-0b46eb26cad5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.873629 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.873597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.873629 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.873635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7d7\" (UniqueName: \"kubernetes.io/projected/59995b9b-a14b-48fd-9844-0b46eb26cad5-kube-api-access-xh7d7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.873872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.873661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.873872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.873679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59995b9b-a14b-48fd-9844-0b46eb26cad5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.873872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.873735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.873872 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.873810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.874117 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.874090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.874242 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.874165    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.874242 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.874196    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.875892 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.875869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.876304 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.876281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59995b9b-a14b-48fd-9844-0b46eb26cad5-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:52.881466 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:52.881447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7d7\" (UniqueName: \"kubernetes.io/projected/59995b9b-a14b-48fd-9844-0b46eb26cad5-kube-api-access-xh7d7\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.326610 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.326571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.331086 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.331059    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.427824 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.427775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.427994 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.427877    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.430694 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.430678    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") DevicePath \"csi-54b13d7209a401aa8c0f2432aea2d571fbb1056184ad7cd1c9a32ca59ae4b7be\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.521043 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.521012    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ab6e994f0711b85347795a4cc0207f4d9c0e47d630289dc945bbd82e3298c27e/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.537729 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.537655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.710471 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.710430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:30:58.856950 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:58.856923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"]
Jul 29 23:30:58.858617 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:30:58.858589    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59995b9b_a14b_48fd_9844_0b46eb26cad5.slice/crio-67bf66e1f5138edec60d5f02efc87622f1b3088ebe900304708c2df88a1c82e4 WatchSource:0}: Error finding container 67bf66e1f5138edec60d5f02efc87622f1b3088ebe900304708c2df88a1c82e4: Status 404 returned error can't find the container with id 67bf66e1f5138edec60d5f02efc87622f1b3088ebe900304708c2df88a1c82e4
Jul 29 23:30:59.427153 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:59.427107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" event={"ID":"59995b9b-a14b-48fd-9844-0b46eb26cad5","Type":"ContainerStarted","Data":"5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393"}
Jul 29 23:30:59.427645 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:59.427177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" event={"ID":"59995b9b-a14b-48fd-9844-0b46eb26cad5","Type":"ContainerStarted","Data":"67bf66e1f5138edec60d5f02efc87622f1b3088ebe900304708c2df88a1c82e4"}
Jul 29 23:30:59.445386 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:59.445338    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podStartSLOduration=7.445321412 podStartE2EDuration="7.445321412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 23:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:30:59.444016679 +0000 UTC m=+2675.352707073" watchObservedRunningTime="2026-07-29 23:30:59.445321412 +0000 UTC m=+2675.354011772"
Jul 29 23:30:59.711646 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:30:59.711533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:31:08.711246 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:08.711205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:31:08.713133 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:08.713095    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:31:18.711704 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:18.711605    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:31:28.711838 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:28.711758    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:31:38.711016 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:38.710974    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:31:48.711579 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:48.711528    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:31:58.711498 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:31:58.711448    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:32:08.711519 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:08.711468    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:32:18.711232 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:18.711186    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:32:28.711310 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:28.711262    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:32:38.711926 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:38.711878    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:32:48.711888 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:48.711759    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 29 23:32:58.720903 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:58.720869    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:32:58.729100 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:32:58.729072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:33:03.707658 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:03.707621    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"]
Jul 29 23:33:18.733051 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:18.733008    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main" containerID="cri-o://5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393" gracePeriod=45
Jul 29 23:33:19.489467 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.489442    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:33:19.585973 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.585941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-home\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586168 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.585985    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-tmp-dir\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586168 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59995b9b-a14b-48fd-9844-0b46eb26cad5-tls-certs\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586168 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586033    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-model-cache\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586168 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586413 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586185    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-dshm\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586413 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586216    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh7d7\" (UniqueName: \"kubernetes.io/projected/59995b9b-a14b-48fd-9844-0b46eb26cad5-kube-api-access-xh7d7\") pod \"59995b9b-a14b-48fd-9844-0b46eb26cad5\" (UID: \"59995b9b-a14b-48fd-9844-0b46eb26cad5\") "
Jul 29 23:33:19.586413 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586318    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-model-cache" (OuterVolumeSpecName: "model-cache") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:33:19.586575 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586504    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.586809 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.586746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-home" (OuterVolumeSpecName: "home") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:33:19.588498 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.588468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59995b9b-a14b-48fd-9844-0b46eb26cad5-kube-api-access-xh7d7" (OuterVolumeSpecName: "kube-api-access-xh7d7") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "kube-api-access-xh7d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:33:19.588683 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.588660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59995b9b-a14b-48fd-9844-0b46eb26cad5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:33:19.588683 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.588674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-dshm" (OuterVolumeSpecName: "dshm") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:33:19.589085 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.589065    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e" (OuterVolumeSpecName: "kserve-pvc-source") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 23:33:19.603417 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.603385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59995b9b-a14b-48fd-9844-0b46eb26cad5" (UID: "59995b9b-a14b-48fd-9844-0b46eb26cad5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:33:19.687607 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.687570    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59995b9b-a14b-48fd-9844-0b46eb26cad5-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.687772 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.687620    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") on node \"ip-10-0-131-193.ec2.internal\" "
Jul 29 23:33:19.687772 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.687630    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.687772 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.687639    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7d7\" (UniqueName: \"kubernetes.io/projected/59995b9b-a14b-48fd-9844-0b46eb26cad5-kube-api-access-xh7d7\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.687772 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.687649    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.687772 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.687657    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59995b9b-a14b-48fd-9844-0b46eb26cad5-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:19.715551 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.715532    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e") on node "ip-10-0-131-193.ec2.internal"
Jul 29 23:33:19.788096 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:19.788064    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-e849e7f6-9360-4d14-8ff5-cfadd8d71538\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e3648de10abafa7e\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"csi-54b13d7209a401aa8c0f2432aea2d571fbb1056184ad7cd1c9a32ca59ae4b7be\""
Jul 29 23:33:20.234312 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.234280    2580 generic.go:356] "Generic (PLEG): container finished" podID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerID="5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393" exitCode=0
Jul 29 23:33:20.234496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.234326    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"
Jul 29 23:33:20.234496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.234337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" event={"ID":"59995b9b-a14b-48fd-9844-0b46eb26cad5","Type":"ContainerDied","Data":"5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393"}
Jul 29 23:33:20.234496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.234458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2" event={"ID":"59995b9b-a14b-48fd-9844-0b46eb26cad5","Type":"ContainerDied","Data":"67bf66e1f5138edec60d5f02efc87622f1b3088ebe900304708c2df88a1c82e4"}
Jul 29 23:33:20.234496 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.234485    2580 scope.go:117] "RemoveContainer" containerID="5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393"
Jul 29 23:33:20.243973 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.243954    2580 scope.go:117] "RemoveContainer" containerID="5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393"
Jul 29 23:33:20.244223 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:33:20.244205    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393\": container with ID starting with 5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393 not found: ID does not exist" containerID="5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393"
Jul 29 23:33:20.244285 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.244230    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393"} err="failed to get container status \"5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393\": rpc error: code = NotFound desc = could not find container \"5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393\": container with ID starting with 5aa388e4b9eaee2b0efa1a0a9faf6452014ae82c0dbb48e06804de3745114393 not found: ID does not exist"
Jul 29 23:33:20.254832 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.254808    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"]
Jul 29 23:33:20.260463 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.260442    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcv7dz2"]
Jul 29 23:33:20.569698 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:20.569663    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" path="/var/lib/kubelet/pods/59995b9b-a14b-48fd-9844-0b46eb26cad5/volumes"
Jul 29 23:33:26.905919 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.905880    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"]
Jul 29 23:33:26.906816 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.906772    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main"
Jul 29 23:33:26.906816 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.906813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main"
Jul 29 23:33:26.907040 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.907027    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59995b9b-a14b-48fd-9844-0b46eb26cad5" containerName="main"
Jul 29 23:33:26.912762 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.912732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:26.915026 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.914998    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 23:33:26.915178 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.915023    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 23:33:26.915341 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.915323    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-sdqb6\""
Jul 29 23:33:26.915596 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.915550    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"]
Jul 29 23:33:26.948665 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:26.948638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xw6\" (UniqueName: \"kubernetes.io/projected/888a1a32-eb0b-4d1d-ac57-bc0e20896236-kube-api-access-d9xw6\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:27.049435 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:27.049396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xw6\" (UniqueName: \"kubernetes.io/projected/888a1a32-eb0b-4d1d-ac57-bc0e20896236-kube-api-access-d9xw6\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:27.057121 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:27.057089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xw6\" (UniqueName: \"kubernetes.io/projected/888a1a32-eb0b-4d1d-ac57-bc0e20896236-kube-api-access-d9xw6\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:28.625167 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:28.625138    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"]
Jul 29 23:33:28.629060 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:28.629035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6kdhv"]
Jul 29 23:33:30.569819 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:30.569768    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8af150cd-149b-4dbb-a901-f991ea198969" path="/var/lib/kubelet/pods/8af150cd-149b-4dbb-a901-f991ea198969/volumes"
Jul 29 23:33:32.195126 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.195085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.199963 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.199938    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.296341 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.296307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.296527 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.296422    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.299372 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.299351    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") DevicePath \"csi-432cb4c176574272915f299d2e42849927080919195d4868bd96f29d960cb449\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.528982 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.528946    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b7f2771dbbf4dc933fab169ff4eaab107407536e7d9057f466ad95e87a5c92de/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.542381 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.542352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"e2e-pvc-model-download-dfd2l\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.624286 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.624253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:32.752231 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:32.752203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"]
Jul 29 23:33:32.754153 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:33:32.754125    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888a1a32_eb0b_4d1d_ac57_bc0e20896236.slice/crio-f2cf2f1e9ec6946b68ccb9926746e2c3f16f8028a90c069904def8acb49ca3ec WatchSource:0}: Error finding container f2cf2f1e9ec6946b68ccb9926746e2c3f16f8028a90c069904def8acb49ca3ec: Status 404 returned error can't find the container with id f2cf2f1e9ec6946b68ccb9926746e2c3f16f8028a90c069904def8acb49ca3ec
Jul 29 23:33:33.303708 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:33.303675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l" event={"ID":"888a1a32-eb0b-4d1d-ac57-bc0e20896236","Type":"ContainerStarted","Data":"d29ca9a3929336f7024b97303e08bb1500b4087293bd1d30db3f781ce7a6657a"}
Jul 29 23:33:33.303708 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:33.303716    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l" event={"ID":"888a1a32-eb0b-4d1d-ac57-bc0e20896236","Type":"ContainerStarted","Data":"f2cf2f1e9ec6946b68ccb9926746e2c3f16f8028a90c069904def8acb49ca3ec"}
Jul 29 23:33:33.318219 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:33.318170    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l" podStartSLOduration=10.318154449 podStartE2EDuration="10.318154449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 23:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:33:33.31720801 +0000 UTC m=+2829.225898367" watchObservedRunningTime="2026-07-29 23:33:33.318154449 +0000 UTC m=+2829.226844807"
Jul 29 23:33:40.345663 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:40.345629    2580 generic.go:356] "Generic (PLEG): container finished" podID="888a1a32-eb0b-4d1d-ac57-bc0e20896236" containerID="d29ca9a3929336f7024b97303e08bb1500b4087293bd1d30db3f781ce7a6657a" exitCode=0
Jul 29 23:33:40.346137 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:40.345701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l" event={"ID":"888a1a32-eb0b-4d1d-ac57-bc0e20896236","Type":"ContainerDied","Data":"d29ca9a3929336f7024b97303e08bb1500b4087293bd1d30db3f781ce7a6657a"}
Jul 29 23:33:41.504584 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.504560    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:41.594128 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.594087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9xw6\" (UniqueName: \"kubernetes.io/projected/888a1a32-eb0b-4d1d-ac57-bc0e20896236-kube-api-access-d9xw6\") pod \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") "
Jul 29 23:33:41.594304 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.594251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\" (UID: \"888a1a32-eb0b-4d1d-ac57-bc0e20896236\") "
Jul 29 23:33:41.596437 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.596403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888a1a32-eb0b-4d1d-ac57-bc0e20896236-kube-api-access-d9xw6" (OuterVolumeSpecName: "kube-api-access-d9xw6") pod "888a1a32-eb0b-4d1d-ac57-bc0e20896236" (UID: "888a1a32-eb0b-4d1d-ac57-bc0e20896236"). InnerVolumeSpecName "kube-api-access-d9xw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:33:41.596999 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.596978    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd" (OuterVolumeSpecName: "model-storage") pod "888a1a32-eb0b-4d1d-ac57-bc0e20896236" (UID: "888a1a32-eb0b-4d1d-ac57-bc0e20896236"). InnerVolumeSpecName "pvc-c2043d33-7945-4454-be43-31b1a1e806e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 23:33:41.695469 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.695375    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xw6\" (UniqueName: \"kubernetes.io/projected/888a1a32-eb0b-4d1d-ac57-bc0e20896236-kube-api-access-d9xw6\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:33:41.695469 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.695420    2580 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") on node \"ip-10-0-131-193.ec2.internal\" "
Jul 29 23:33:41.813216 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.813195    2580 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-c2043d33-7945-4454-be43-31b1a1e806e0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd") on node "ip-10-0-131-193.ec2.internal"
Jul 29 23:33:41.897336 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:41.897292    2580 reconciler_common.go:299] "Volume detached for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"csi-432cb4c176574272915f299d2e42849927080919195d4868bd96f29d960cb449\""
Jul 29 23:33:42.357465 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:42.357436    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l"
Jul 29 23:33:42.357649 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:42.357482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-dfd2l" event={"ID":"888a1a32-eb0b-4d1d-ac57-bc0e20896236","Type":"ContainerDied","Data":"f2cf2f1e9ec6946b68ccb9926746e2c3f16f8028a90c069904def8acb49ca3ec"}
Jul 29 23:33:42.357649 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:42.357516    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2cf2f1e9ec6946b68ccb9926746e2c3f16f8028a90c069904def8acb49ca3ec"
Jul 29 23:33:59.528536 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.528505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"]
Jul 29 23:33:59.528966 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.528865    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888a1a32-eb0b-4d1d-ac57-bc0e20896236" containerName="storage-initializer"
Jul 29 23:33:59.528966 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.528877    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="888a1a32-eb0b-4d1d-ac57-bc0e20896236" containerName="storage-initializer"
Jul 29 23:33:59.528966 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.528954    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="888a1a32-eb0b-4d1d-ac57-bc0e20896236" containerName="storage-initializer"
Jul 29 23:33:59.532125 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.532103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.534812 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.534535    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 23:33:59.534812 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.534655    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 23:33:59.534812 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.534672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 29 23:33:59.534812 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.534679    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-sdqb6\""
Jul 29 23:33:59.534812 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.534748    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5lxhk\""
Jul 29 23:33:59.546374 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.546351    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"]
Jul 29 23:33:59.647344 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.647312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.647344 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.647355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.647611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.647376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96zt\" (UniqueName: \"kubernetes.io/projected/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-kube-api-access-t96zt\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.647611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.647395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.647611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.647524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.647611 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.647568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.748471 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.748657 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.748657 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96zt\" (UniqueName: \"kubernetes.io/projected/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-kube-api-access-t96zt\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.748657 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.748657 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.749134 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748950    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.749134 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.748999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.749134 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.749043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.749134 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.749086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.751154 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.751131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.751518 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.751495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:33:59.763673 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:33:59.763646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96zt\" (UniqueName: \"kubernetes.io/projected/a503c2b0-6dfe-49fb-8089-5bd8d0b7f302-kube-api-access-t96zt\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.775297 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.775261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.779429 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.779405    2580 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.876368 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.876330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.876554 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.876461    2580 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.878723 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.878701    2580 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") DevicePath \"csi-432cb4c176574272915f299d2e42849927080919195d4868bd96f29d960cb449\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.966075 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.966034    2580 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b7f2771dbbf4dc933fab169ff4eaab107407536e7d9057f466ad95e87a5c92de/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:02.984962 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:02.984935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-c2043d33-7945-4454-be43-31b1a1e806e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e7cb704bac01e6cd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl\" (UID: \"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:03.144945 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:03.144908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:03.287310 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:03.287287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"]
Jul 29 23:34:03.289102 ip-10-0-131-193 kubenswrapper[2580]: W0729 23:34:03.289073    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda503c2b0_6dfe_49fb_8089_5bd8d0b7f302.slice/crio-501eac18d0ac80df1c83ec80b998bca13ff037536a3823a9a580f139a6dc7f5c WatchSource:0}: Error finding container 501eac18d0ac80df1c83ec80b998bca13ff037536a3823a9a580f139a6dc7f5c: Status 404 returned error can't find the container with id 501eac18d0ac80df1c83ec80b998bca13ff037536a3823a9a580f139a6dc7f5c
Jul 29 23:34:03.468226 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:03.468085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" event={"ID":"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302","Type":"ContainerStarted","Data":"d2c172a4a5f357599999c550e72fee55324c6324e5b32f9cbba71b61ba135e9a"}
Jul 29 23:34:03.468226 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:03.468160    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:03.468226 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:03.468183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" event={"ID":"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302","Type":"ContainerStarted","Data":"501eac18d0ac80df1c83ec80b998bca13ff037536a3823a9a580f139a6dc7f5c"}
Jul 29 23:34:04.476507 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:04.476465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" event={"ID":"a503c2b0-6dfe-49fb-8089-5bd8d0b7f302","Type":"ContainerStarted","Data":"9b22d2a4d56aa1496299c4f533473547b5e862d42abb19e77732c80968269a11"}
Jul 29 23:34:04.506473 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:04.506413    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podStartSLOduration=5.506389884 podStartE2EDuration="5.506389884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 23:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 23:34:04.503914103 +0000 UTC m=+2860.412604461" watchObservedRunningTime="2026-07-29 23:34:04.506389884 +0000 UTC m=+2860.415080243"
Jul 29 23:34:13.145339 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:13.145300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:13.145953 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:13.145379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:13.146818 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:13.146767    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:34:13.159895 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:13.159871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:34:23.145659 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:23.145555    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:34:33.145603 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:33.145556    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:34:43.145543 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:43.145487    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:34:53.145807 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:34:53.145745    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:35:03.146340 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:35:03.146285    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:35:13.146017 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:35:13.145966    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:35:23.146363 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:35:23.146304    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:35:33.145813 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:35:33.145740    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:35:43.146189 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:35:43.146138    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:35:53.145931 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:35:53.145814    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl" podUID="a503c2b0-6dfe-49fb-8089-5bd8d0b7f302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 29 23:36:03.155569 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:36:03.155533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:36:03.171326 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:36:03.171301    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl"
Jul 29 23:37:25.741924 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:37:25.741841    2580 scope.go:117] "RemoveContainer" containerID="79ba4949a3b880b74330c9547bf7a2034ed280b3e680076424a99bcbc528a1f7"
Jul 29 23:48:44.773375 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:48:44.773340    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl_a503c2b0-6dfe-49fb-8089-5bd8d0b7f302/llm-d-routing-sidecar/0.log"
Jul 29 23:48:44.781975 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:48:44.781944    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-86c954499fcgljl_a503c2b0-6dfe-49fb-8089-5bd8d0b7f302/main/0.log"
Jul 29 23:49:18.516135 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.516050    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 23:49:18.519594 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.519576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.524925 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.524881    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 29 23:49:18.525108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.525015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 23:49:18.525108 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.525041    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 23:49:18.525240 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.525159    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-fh7hr\""
Jul 29 23:49:18.536929 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.536901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 23:49:18.537172 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.537157    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-vxspr\""
Jul 29 23:49:18.615581 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.615546    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.615759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.615661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.615759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.615705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.615759 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.615734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.615906 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.615769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.716540 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.716721 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.716721 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.716721 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.716915 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.716915 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716846    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cnc\" (UniqueName: \"kubernetes.io/projected/899c48f1-02cf-47c5-84bd-c66678a33193-kube-api-access-t4cnc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.717034 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.716920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.717034 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.717003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.717159 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.717134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.718893 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.718873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.719332 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.719308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.817367 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.817324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cnc\" (UniqueName: \"kubernetes.io/projected/899c48f1-02cf-47c5-84bd-c66678a33193-kube-api-access-t4cnc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:49:18.829006 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:49:18.828980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cnc\" (UniqueName: \"kubernetes.io/projected/899c48f1-02cf-47c5-84bd-c66678a33193-kube-api-access-t4cnc\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:51:21.530578 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:21.530531    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="899c48f1-02cf-47c5-84bd-c66678a33193"
Jul 29 23:51:22.129810 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:22.129755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:51:22.231346 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:22.231311    2580 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:22.231524 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:22.231407    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs podName:899c48f1-02cf-47c5-84bd-c66678a33193 nodeName:}" failed. No retries permitted until 2026-07-29 23:51:22.731387089 +0000 UTC m=+3898.640077445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:22.736826 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:22.736769    2580 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:22.737239 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:22.736864    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs podName:899c48f1-02cf-47c5-84bd-c66678a33193 nodeName:}" failed. No retries permitted until 2026-07-29 23:51:23.736847573 +0000 UTC m=+3899.645537916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:23.747060 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:23.747019    2580 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:23.747579 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:23.747116    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs podName:899c48f1-02cf-47c5-84bd-c66678a33193 nodeName:}" failed. No retries permitted until 2026-07-29 23:51:25.747099294 +0000 UTC m=+3901.655789631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:25.766514 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:25.766476    2580 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:25.766929 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:25.766550    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs podName:899c48f1-02cf-47c5-84bd-c66678a33193 nodeName:}" failed. No retries permitted until 2026-07-29 23:51:29.766536536 +0000 UTC m=+3905.675226872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:29.802719 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:29.802685    2580 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-7c3633d1/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:29.803179 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:29.802762    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs podName:899c48f1-02cf-47c5-84bd-c66678a33193 nodeName:}" failed. No retries permitted until 2026-07-29 23:51:37.802746438 +0000 UTC m=+3913.711436779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 29 23:51:30.980997 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:30.980958    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 23:51:30.981386 ip-10-0-131-193 kubenswrapper[2580]: E0729 23:51:30.981252    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context canceled" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="899c48f1-02cf-47c5-84bd-c66678a33193"
Jul 29 23:51:31.177365 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.177335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:51:31.184025 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.184002    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:51:31.216018 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.215990    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-tmp-dir\") pod \"899c48f1-02cf-47c5-84bd-c66678a33193\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") "
Jul 29 23:51:31.216116 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216034    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs\") pod \"899c48f1-02cf-47c5-84bd-c66678a33193\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") "
Jul 29 23:51:31.216116 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-home\") pod \"899c48f1-02cf-47c5-84bd-c66678a33193\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") "
Jul 29 23:51:31.216116 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216101    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-dshm\") pod \"899c48f1-02cf-47c5-84bd-c66678a33193\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") "
Jul 29 23:51:31.216285 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4cnc\" (UniqueName: \"kubernetes.io/projected/899c48f1-02cf-47c5-84bd-c66678a33193-kube-api-access-t4cnc\") pod \"899c48f1-02cf-47c5-84bd-c66678a33193\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") "
Jul 29 23:51:31.216285 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216172    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-model-cache\") pod \"899c48f1-02cf-47c5-84bd-c66678a33193\" (UID: \"899c48f1-02cf-47c5-84bd-c66678a33193\") "
Jul 29 23:51:31.216285 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "899c48f1-02cf-47c5-84bd-c66678a33193" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:31.216441 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216352    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-home" (OuterVolumeSpecName: "home") pod "899c48f1-02cf-47c5-84bd-c66678a33193" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:31.216516 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-model-cache" (OuterVolumeSpecName: "model-cache") pod "899c48f1-02cf-47c5-84bd-c66678a33193" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:31.216563 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216501    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:51:31.216563 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.216541    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:51:31.218152 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.218123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-dshm" (OuterVolumeSpecName: "dshm") pod "899c48f1-02cf-47c5-84bd-c66678a33193" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 23:51:31.218247 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.218215    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "899c48f1-02cf-47c5-84bd-c66678a33193" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 23:51:31.218294 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.218254    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899c48f1-02cf-47c5-84bd-c66678a33193-kube-api-access-t4cnc" (OuterVolumeSpecName: "kube-api-access-t4cnc") pod "899c48f1-02cf-47c5-84bd-c66678a33193" (UID: "899c48f1-02cf-47c5-84bd-c66678a33193"). InnerVolumeSpecName "kube-api-access-t4cnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 23:51:31.317599 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.317576    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899c48f1-02cf-47c5-84bd-c66678a33193-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:51:31.317599 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.317599    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:51:31.317781 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.317609    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cnc\" (UniqueName: \"kubernetes.io/projected/899c48f1-02cf-47c5-84bd-c66678a33193-kube-api-access-t4cnc\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:51:31.317781 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:31.317619    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899c48f1-02cf-47c5-84bd-c66678a33193-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 29 23:51:32.182552 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:32.182519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 29 23:51:32.213487 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:32.213454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 23:51:32.219307 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:32.219279    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 29 23:51:32.571925 ip-10-0-131-193 kubenswrapper[2580]: I0729 23:51:32.571895    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899c48f1-02cf-47c5-84bd-c66678a33193" path="/var/lib/kubelet/pods/899c48f1-02cf-47c5-84bd-c66678a33193/volumes"
Jul 30 00:00:00.143022 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.142981    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29756160-sjnbv"]
Jul 30 00:00:00.146448 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.146429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.148487 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.148460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Jul 30 00:00:00.148487 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.148476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-wzmtp\""
Jul 30 00:00:00.154506 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.154481    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29756160-sjnbv"]
Jul 30 00:00:00.269644 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.269610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvz4\" (UniqueName: \"kubernetes.io/projected/1f9d8301-e499-4bde-ba27-e89560ea09bf-kube-api-access-8bvz4\") pod \"image-pruner-29756160-sjnbv\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") " pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.269833 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.269728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f9d8301-e499-4bde-ba27-e89560ea09bf-serviceca\") pod \"image-pruner-29756160-sjnbv\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") " pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.371193 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.371158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f9d8301-e499-4bde-ba27-e89560ea09bf-serviceca\") pod \"image-pruner-29756160-sjnbv\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") " pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.371432 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.371243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvz4\" (UniqueName: \"kubernetes.io/projected/1f9d8301-e499-4bde-ba27-e89560ea09bf-kube-api-access-8bvz4\") pod \"image-pruner-29756160-sjnbv\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") " pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.371801 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.371770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f9d8301-e499-4bde-ba27-e89560ea09bf-serviceca\") pod \"image-pruner-29756160-sjnbv\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") " pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.389603 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.389503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvz4\" (UniqueName: \"kubernetes.io/projected/1f9d8301-e499-4bde-ba27-e89560ea09bf-kube-api-access-8bvz4\") pod \"image-pruner-29756160-sjnbv\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") " pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.474621 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.474522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:00.639917 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:00.639890    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29756160-sjnbv"]
Jul 30 00:00:00.641998 ip-10-0-131-193 kubenswrapper[2580]: W0730 00:00:00.641970    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9d8301_e499_4bde_ba27_e89560ea09bf.slice/crio-9cd428490edc71adf768049defc513ae17b17ef78a5befed2d63d716731da791 WatchSource:0}: Error finding container 9cd428490edc71adf768049defc513ae17b17ef78a5befed2d63d716731da791: Status 404 returned error can't find the container with id 9cd428490edc71adf768049defc513ae17b17ef78a5befed2d63d716731da791
Jul 30 00:00:01.020988 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:01.020930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29756160-sjnbv" event={"ID":"1f9d8301-e499-4bde-ba27-e89560ea09bf","Type":"ContainerStarted","Data":"4fb329f3343ebaaee38bd36278b8db73a3b6e54a943b6d9ebfffc0c3b67283bc"}
Jul 30 00:00:01.021227 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:01.021014    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29756160-sjnbv" event={"ID":"1f9d8301-e499-4bde-ba27-e89560ea09bf","Type":"ContainerStarted","Data":"9cd428490edc71adf768049defc513ae17b17ef78a5befed2d63d716731da791"}
Jul 30 00:00:01.036016 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:01.035942    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29756160-sjnbv" podStartSLOduration=1.035924104 podStartE2EDuration="1.035924104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:00:01.035034996 +0000 UTC m=+4416.943725355" watchObservedRunningTime="2026-07-30 00:00:01.035924104 +0000 UTC m=+4416.944614462"
Jul 30 00:00:02.028046 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:02.028013    2580 generic.go:356] "Generic (PLEG): container finished" podID="1f9d8301-e499-4bde-ba27-e89560ea09bf" containerID="4fb329f3343ebaaee38bd36278b8db73a3b6e54a943b6d9ebfffc0c3b67283bc" exitCode=0
Jul 30 00:00:02.028479 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:02.028065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29756160-sjnbv" event={"ID":"1f9d8301-e499-4bde-ba27-e89560ea09bf","Type":"ContainerDied","Data":"4fb329f3343ebaaee38bd36278b8db73a3b6e54a943b6d9ebfffc0c3b67283bc"}
Jul 30 00:00:03.182487 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.182461    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:03.299974 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.299879    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f9d8301-e499-4bde-ba27-e89560ea09bf-serviceca\") pod \"1f9d8301-e499-4bde-ba27-e89560ea09bf\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") "
Jul 30 00:00:03.299974 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.299928    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bvz4\" (UniqueName: \"kubernetes.io/projected/1f9d8301-e499-4bde-ba27-e89560ea09bf-kube-api-access-8bvz4\") pod \"1f9d8301-e499-4bde-ba27-e89560ea09bf\" (UID: \"1f9d8301-e499-4bde-ba27-e89560ea09bf\") "
Jul 30 00:00:03.300358 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.300332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9d8301-e499-4bde-ba27-e89560ea09bf-serviceca" (OuterVolumeSpecName: "serviceca") pod "1f9d8301-e499-4bde-ba27-e89560ea09bf" (UID: "1f9d8301-e499-4bde-ba27-e89560ea09bf"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 00:00:03.302112 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.302080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9d8301-e499-4bde-ba27-e89560ea09bf-kube-api-access-8bvz4" (OuterVolumeSpecName: "kube-api-access-8bvz4") pod "1f9d8301-e499-4bde-ba27-e89560ea09bf" (UID: "1f9d8301-e499-4bde-ba27-e89560ea09bf"). InnerVolumeSpecName "kube-api-access-8bvz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:00:03.401143 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.401109    2580 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f9d8301-e499-4bde-ba27-e89560ea09bf-serviceca\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:00:03.401143 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:03.401140    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvz4\" (UniqueName: \"kubernetes.io/projected/1f9d8301-e499-4bde-ba27-e89560ea09bf-kube-api-access-8bvz4\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:00:04.043201 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:04.043170    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29756160-sjnbv"
Jul 30 00:00:04.043410 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:04.043206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29756160-sjnbv" event={"ID":"1f9d8301-e499-4bde-ba27-e89560ea09bf","Type":"ContainerDied","Data":"9cd428490edc71adf768049defc513ae17b17ef78a5befed2d63d716731da791"}
Jul 30 00:00:04.043410 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:00:04.043257    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cd428490edc71adf768049defc513ae17b17ef78a5befed2d63d716731da791"
Jul 30 00:09:32.584012 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.583975    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"]
Jul 30 00:09:32.584516 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.584354    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9d8301-e499-4bde-ba27-e89560ea09bf" containerName="image-pruner"
Jul 30 00:09:32.584516 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.584364    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9d8301-e499-4bde-ba27-e89560ea09bf" containerName="image-pruner"
Jul 30 00:09:32.584516 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.584424    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9d8301-e499-4bde-ba27-e89560ea09bf" containerName="image-pruner"
Jul 30 00:09:32.586706 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.586689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.588854 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.588803    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 30 00:09:32.589001 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.588858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 30 00:09:32.589001 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.588915    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 30 00:09:32.589001 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.588955    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 30 00:09:32.589001 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.588962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-cpsfq\""
Jul 30 00:09:32.598131 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.598106    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"]
Jul 30 00:09:32.644745 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-home\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.644745 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-tmp-dir\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.644998 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-dshm\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.644998 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-model-cache\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.644998 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094e382d-6220-4bd7-8a55-b73a20e8fc70-tls-certs\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.644998 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-kserve-provision-location\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.644998 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.644908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmz84\" (UniqueName: \"kubernetes.io/projected/094e382d-6220-4bd7-8a55-b73a20e8fc70-kube-api-access-qmz84\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.745991 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.745956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-model-cache\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.745991 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.745997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094e382d-6220-4bd7-8a55-b73a20e8fc70-tls-certs\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746206 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-kserve-provision-location\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746206 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmz84\" (UniqueName: \"kubernetes.io/projected/094e382d-6220-4bd7-8a55-b73a20e8fc70-kube-api-access-qmz84\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746206 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-home\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746206 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-tmp-dir\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746206 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-dshm\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746488 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-kserve-provision-location\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746488 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-model-cache\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746625 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-home\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.746625 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.746536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-tmp-dir\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.748302 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.748280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-dshm\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.748643 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.748627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094e382d-6220-4bd7-8a55-b73a20e8fc70-tls-certs\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.753940 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.753917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmz84\" (UniqueName: \"kubernetes.io/projected/094e382d-6220-4bd7-8a55-b73a20e8fc70-kube-api-access-qmz84\") pod \"prestop-hook-test-kserve-6597646664-5cnvg\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:32.898642 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:32.898550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:33.040627 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:33.040598    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"]
Jul 30 00:09:33.042832 ip-10-0-131-193 kubenswrapper[2580]: W0730 00:09:33.042804    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094e382d_6220_4bd7_8a55_b73a20e8fc70.slice/crio-53e86c867dc646bf64ebdc41268b8fc2e8ecb26c8469f1dfa63489f3ea719d04 WatchSource:0}: Error finding container 53e86c867dc646bf64ebdc41268b8fc2e8ecb26c8469f1dfa63489f3ea719d04: Status 404 returned error can't find the container with id 53e86c867dc646bf64ebdc41268b8fc2e8ecb26c8469f1dfa63489f3ea719d04
Jul 30 00:09:33.293932 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:33.293835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" event={"ID":"094e382d-6220-4bd7-8a55-b73a20e8fc70","Type":"ContainerStarted","Data":"bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6"}
Jul 30 00:09:33.293932 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:33.293901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" event={"ID":"094e382d-6220-4bd7-8a55-b73a20e8fc70","Type":"ContainerStarted","Data":"53e86c867dc646bf64ebdc41268b8fc2e8ecb26c8469f1dfa63489f3ea719d04"}
Jul 30 00:09:38.326055 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:38.326022    2580 generic.go:356] "Generic (PLEG): container finished" podID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerID="bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6" exitCode=0
Jul 30 00:09:38.326458 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:38.326104    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" event={"ID":"094e382d-6220-4bd7-8a55-b73a20e8fc70","Type":"ContainerDied","Data":"bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6"}
Jul 30 00:09:39.333033 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:39.332994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" event={"ID":"094e382d-6220-4bd7-8a55-b73a20e8fc70","Type":"ContainerStarted","Data":"9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0"}
Jul 30 00:09:39.351735 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:39.351680    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" podStartSLOduration=2.482632519 podStartE2EDuration="7.351658644s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.869026125s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:09:39.350654156 +0000 UTC m=+4995.259344516" watchObservedRunningTime="2026-07-30 00:09:39.351658644 +0000 UTC m=+4995.260349004"
Jul 30 00:09:39.899652 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:39.899613    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:42.898983 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:42.898937    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:09:42.900578 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:42.900548    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 30 00:09:52.899200 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:09:52.899151    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 30 00:10:02.899961 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:02.899914    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 30 00:10:12.899555 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:12.899498    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 30 00:10:14.162450 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:14.162410    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"]
Jul 30 00:10:29.186189 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:29.186153    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main" containerID="cri-o://9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0" gracePeriod=45
Jul 30 00:10:31.048639 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.048615    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6597646664-5cnvg_094e382d-6220-4bd7-8a55-b73a20e8fc70/main/0.log"
Jul 30 00:10:31.050392 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.050375    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:10:31.102671 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-kserve-provision-location\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.102830 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-dshm\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.102830 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102720    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-home\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.102830 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102777    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmz84\" (UniqueName: \"kubernetes.io/projected/094e382d-6220-4bd7-8a55-b73a20e8fc70-kube-api-access-qmz84\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.103018 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094e382d-6220-4bd7-8a55-b73a20e8fc70-tls-certs\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.103018 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-tmp-dir\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.103018 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.102889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-model-cache\") pod \"094e382d-6220-4bd7-8a55-b73a20e8fc70\" (UID: \"094e382d-6220-4bd7-8a55-b73a20e8fc70\") "
Jul 30 00:10:31.103188 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.103014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-home" (OuterVolumeSpecName: "home") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:10:31.103188 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.103159    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.103286 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.103192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-model-cache" (OuterVolumeSpecName: "model-cache") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:10:31.103384 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.103356    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:10:31.105039 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.105011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-dshm" (OuterVolumeSpecName: "dshm") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:10:31.105141 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.105105    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e382d-6220-4bd7-8a55-b73a20e8fc70-kube-api-access-qmz84" (OuterVolumeSpecName: "kube-api-access-qmz84") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "kube-api-access-qmz84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:10:31.105477 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.105457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094e382d-6220-4bd7-8a55-b73a20e8fc70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:10:31.172391 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.172355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "094e382d-6220-4bd7-8a55-b73a20e8fc70" (UID: "094e382d-6220-4bd7-8a55-b73a20e8fc70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:10:31.204223 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.204201    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmz84\" (UniqueName: \"kubernetes.io/projected/094e382d-6220-4bd7-8a55-b73a20e8fc70-kube-api-access-qmz84\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.204223 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.204223    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094e382d-6220-4bd7-8a55-b73a20e8fc70-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.204374 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.204232    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.204374 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.204241    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.204374 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.204250    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-kserve-provision-location\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.204374 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.204257    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094e382d-6220-4bd7-8a55-b73a20e8fc70-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:10:31.653306 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.653277    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6597646664-5cnvg_094e382d-6220-4bd7-8a55-b73a20e8fc70/main/0.log"
Jul 30 00:10:31.653713 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.653691    2580 generic.go:356] "Generic (PLEG): container finished" podID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerID="9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0" exitCode=130
Jul 30 00:10:31.653850 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.653740    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"
Jul 30 00:10:31.653850 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.653747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" event={"ID":"094e382d-6220-4bd7-8a55-b73a20e8fc70","Type":"ContainerDied","Data":"9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0"}
Jul 30 00:10:31.653850 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.653820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg" event={"ID":"094e382d-6220-4bd7-8a55-b73a20e8fc70","Type":"ContainerDied","Data":"53e86c867dc646bf64ebdc41268b8fc2e8ecb26c8469f1dfa63489f3ea719d04"}
Jul 30 00:10:31.653995 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.653854    2580 scope.go:117] "RemoveContainer" containerID="9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0"
Jul 30 00:10:31.663863 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.663843    2580 scope.go:117] "RemoveContainer" containerID="bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6"
Jul 30 00:10:31.672265 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.672247    2580 scope.go:117] "RemoveContainer" containerID="9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0"
Jul 30 00:10:31.672550 ip-10-0-131-193 kubenswrapper[2580]: E0730 00:10:31.672531    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0\": container with ID starting with 9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0 not found: ID does not exist" containerID="9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0"
Jul 30 00:10:31.672615 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.672556    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0"} err="failed to get container status \"9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0\": rpc error: code = NotFound desc = could not find container \"9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0\": container with ID starting with 9681582c4a5f459e9af2a3d54856705ae63bf26a86f2db89bdb85aa1e76aeff0 not found: ID does not exist"
Jul 30 00:10:31.672615 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.672575    2580 scope.go:117] "RemoveContainer" containerID="bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6"
Jul 30 00:10:31.672905 ip-10-0-131-193 kubenswrapper[2580]: E0730 00:10:31.672877    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6\": container with ID starting with bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6 not found: ID does not exist" containerID="bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6"
Jul 30 00:10:31.672974 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.672914    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6"} err="failed to get container status \"bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6\": rpc error: code = NotFound desc = could not find container \"bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6\": container with ID starting with bc3ff60a41dcc8a8caa270d7819a1779205ce6cf116118e5f6cbe30a913113b6 not found: ID does not exist"
Jul 30 00:10:31.676444 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.676398    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"]
Jul 30 00:10:31.680191 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:31.680171    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6597646664-5cnvg"]
Jul 30 00:10:32.570432 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:10:32.570396    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" path="/var/lib/kubelet/pods/094e382d-6220-4bd7-8a55-b73a20e8fc70/volumes"
Jul 30 00:11:42.993908 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.993868    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"]
Jul 30 00:11:42.994350 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.994265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="storage-initializer"
Jul 30 00:11:42.994350 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.994278    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="storage-initializer"
Jul 30 00:11:42.994350 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.994287    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main"
Jul 30 00:11:42.994350 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.994292    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main"
Jul 30 00:11:42.994494 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.994372    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="094e382d-6220-4bd7-8a55-b73a20e8fc70" containerName="main"
Jul 30 00:11:42.997604 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.997585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:42.999490 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.999464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 30 00:11:42.999640 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:42.999553    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 30 00:11:43.000226 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.000210    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-2cct7\""
Jul 30 00:11:43.000311 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.000254    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 30 00:11:43.006894 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.006870    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"]
Jul 30 00:11:43.152737 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.152698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.152952 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.152749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.152952 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.152878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.152952 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.152910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.153082 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.152956    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txq4k\" (UniqueName: \"kubernetes.io/projected/7739187e-329c-42b9-a67c-cb30c7bd4cf5-kube-api-access-txq4k\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.153082 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.153033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254154 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254154 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254154 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txq4k\" (UniqueName: \"kubernetes.io/projected/7739187e-329c-42b9-a67c-cb30c7bd4cf5-kube-api-access-txq4k\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254456 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254456 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254456 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254618 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254571    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254618 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.254716 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.254630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.256583 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.256560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.257048 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.257030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.261285 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.261261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txq4k\" (UniqueName: \"kubernetes.io/projected/7739187e-329c-42b9-a67c-cb30c7bd4cf5-kube-api-access-txq4k\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-6zsk4\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.309241 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.309201    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:43.446051 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:43.446025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"]
Jul 30 00:11:44.061943 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:44.061902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" event={"ID":"7739187e-329c-42b9-a67c-cb30c7bd4cf5","Type":"ContainerStarted","Data":"a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"}
Jul 30 00:11:44.062479 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:44.061958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" event={"ID":"7739187e-329c-42b9-a67c-cb30c7bd4cf5","Type":"ContainerStarted","Data":"f70cd930d655c810290a4be7b595ffa33a48c8914b080592544386ebf75f9895"}
Jul 30 00:11:44.079103 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:44.079044    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" podStartSLOduration=2.079028748 podStartE2EDuration="2.079028748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:11:44.077310767 +0000 UTC m=+5119.986001126" watchObservedRunningTime="2026-07-30 00:11:44.079028748 +0000 UTC m=+5119.987719157"
Jul 30 00:11:44.309899 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:44.309862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:53.309554 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:53.309515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:53.322288 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:53.322265    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:54.131617 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:54.131590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:56.917845 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:56.917800    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"]
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]: E0730 00:11:56.928237    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:  > containerID="a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410" cmd=["/bin/sleep","15"]
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:56.928292    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" message="" err=<
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:  >
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]: E0730 00:11:56.928304    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Jul 30 00:11:56.928299 ip-10-0-131-193 kubenswrapper[2580]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerName="main" containerID="cri-o://a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"
Jul 30 00:11:56.929063 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:56.928349    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerName="main" containerID="cri-o://a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410" gracePeriod=60
Jul 30 00:11:57.121572 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.121533    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 30 00:11:57.295100 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.295073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:57.377813 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.377758    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txq4k\" (UniqueName: \"kubernetes.io/projected/7739187e-329c-42b9-a67c-cb30c7bd4cf5-kube-api-access-txq4k\") pod \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") "
Jul 30 00:11:57.377967 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.377851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-dshm\") pod \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") "
Jul 30 00:11:57.377967 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.377890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tls-certs\") pod \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") "
Jul 30 00:11:57.377967 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.377924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-home\") pod \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") "
Jul 30 00:11:57.377967 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.377949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-model-cache\") pod \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") "
Jul 30 00:11:57.378200 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tmp-dir\") pod \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\" (UID: \"7739187e-329c-42b9-a67c-cb30c7bd4cf5\") "
Jul 30 00:11:57.378265 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-home" (OuterVolumeSpecName: "home") pod "7739187e-329c-42b9-a67c-cb30c7bd4cf5" (UID: "7739187e-329c-42b9-a67c-cb30c7bd4cf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:57.378409 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7739187e-329c-42b9-a67c-cb30c7bd4cf5" (UID: "7739187e-329c-42b9-a67c-cb30c7bd4cf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:57.378409 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7739187e-329c-42b9-a67c-cb30c7bd4cf5" (UID: "7739187e-329c-42b9-a67c-cb30c7bd4cf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:57.378613 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378502    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tmp-dir\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:11:57.378613 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378524    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-home\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:11:57.378613 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.378538    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-model-cache\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:11:57.380266 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.380143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-dshm" (OuterVolumeSpecName: "dshm") pod "7739187e-329c-42b9-a67c-cb30c7bd4cf5" (UID: "7739187e-329c-42b9-a67c-cb30c7bd4cf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 00:11:57.380266 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.380221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7739187e-329c-42b9-a67c-cb30c7bd4cf5" (UID: "7739187e-329c-42b9-a67c-cb30c7bd4cf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 00:11:57.380266 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.380240    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7739187e-329c-42b9-a67c-cb30c7bd4cf5-kube-api-access-txq4k" (OuterVolumeSpecName: "kube-api-access-txq4k") pod "7739187e-329c-42b9-a67c-cb30c7bd4cf5" (UID: "7739187e-329c-42b9-a67c-cb30c7bd4cf5"). InnerVolumeSpecName "kube-api-access-txq4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 00:11:57.479870 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.479769    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txq4k\" (UniqueName: \"kubernetes.io/projected/7739187e-329c-42b9-a67c-cb30c7bd4cf5-kube-api-access-txq4k\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:11:57.479870 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.479812    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7739187e-329c-42b9-a67c-cb30c7bd4cf5-dshm\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:11:57.479870 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:57.479829    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7739187e-329c-42b9-a67c-cb30c7bd4cf5-tls-certs\") on node \"ip-10-0-131-193.ec2.internal\" DevicePath \"\""
Jul 30 00:11:58.145558 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.145525    2580 generic.go:356] "Generic (PLEG): container finished" podID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerID="a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410" exitCode=0
Jul 30 00:11:58.146099 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.145578    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"
Jul 30 00:11:58.146099 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.145582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" event={"ID":"7739187e-329c-42b9-a67c-cb30c7bd4cf5","Type":"ContainerDied","Data":"a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"}
Jul 30 00:11:58.146099 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.145640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4" event={"ID":"7739187e-329c-42b9-a67c-cb30c7bd4cf5","Type":"ContainerDied","Data":"f70cd930d655c810290a4be7b595ffa33a48c8914b080592544386ebf75f9895"}
Jul 30 00:11:58.146099 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.145673    2580 scope.go:117] "RemoveContainer" containerID="a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"
Jul 30 00:11:58.155636 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.155615    2580 scope.go:117] "RemoveContainer" containerID="a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"
Jul 30 00:11:58.155936 ip-10-0-131-193 kubenswrapper[2580]: E0730 00:11:58.155914    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410\": container with ID starting with a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410 not found: ID does not exist" containerID="a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"
Jul 30 00:11:58.156001 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.155945    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410"} err="failed to get container status \"a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410\": rpc error: code = NotFound desc = could not find container \"a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410\": container with ID starting with a430368e02f1e177d6875b623b856798c2d6000576d02ff384fe61ecc3247410 not found: ID does not exist"
Jul 30 00:11:58.169154 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.169133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"]
Jul 30 00:11:58.172497 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.172477    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-6zsk4"]
Jul 30 00:11:58.569977 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:11:58.569939    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" path="/var/lib/kubelet/pods/7739187e-329c-42b9-a67c-cb30c7bd4cf5/volumes"
Jul 30 00:13:57.164767 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:57.164737    2580 ???:1] "http: TLS handshake error from 10.0.135.98:34838: EOF"
Jul 30 00:13:57.377907 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:57.377873    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:13:57.391021 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:57.390985    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:13:57.416037 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:57.415961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:13:57.430865 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:57.430817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:13:58.268526 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:58.268477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:13:58.290246 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:58.290222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:13:58.313539 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:58.313516    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:13:58.328070 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:58.328047    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:13:59.182025 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:59.181985    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:13:59.198363 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:59.198333    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:13:59.231519 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:59.231488    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:13:59.244586 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:13:59.244562    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:14:00.064441 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:00.064412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:14:00.081288 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:00.081259    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:14:00.103604 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:00.103575    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:14:00.119068 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:00.119048    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:14:00.910173 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:00.910134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:00.923015 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:00.922986    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:01.096480 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:01.096446    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:01.105707 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:01.105679    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:02.062333 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:02.062288    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:02.091652 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:02.091602    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:02.267767 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:02.267735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:02.336757 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:02.336651    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:03.718350 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:03.718282    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:03.731091 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:03.731066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:03.884108 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:03.884074    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:03.896686 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:03.896663    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:04.735591 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:04.735564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:04.749345 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:04.749324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:04.904639 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:04.904606    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:04.918359 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:04.918331    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:05.766074 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:05.766043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:05.779932 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:05.779905    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:05.936855 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:05.936771    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:05.949680 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:05.949649    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:06.798505 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:06.798475    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:06.812219 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:06.812191    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:06.962401 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:06.962353    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:06.974682 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:06.974645    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:07.795355 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:07.795310    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:07.807954 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:07.807918    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:07.959801 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:07.959768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:07.972590 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:07.972566    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:08.797154 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:08.797122    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:08.810019 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:08.809992    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:08.961771 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:08.961735    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:08.974295 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:08.974271    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:09.810664 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:09.810639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:09.826348 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:09.826325    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:09.981632 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:09.981599    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:09.993553 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:09.993526    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:10.821469 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:10.821411    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:10.834130 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:10.834075    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:10.997944 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:10.997910    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:11.010383 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:11.010356    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:11.860026 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:11.859998    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:11.871524 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:11.871501    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:12.041289 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:12.041260    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:12.058017 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:12.057993    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:12.884215 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:12.884185    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:12.896693 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:12.896665    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:13.043964 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:13.043931    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:13.057498 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:13.057472    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:13.931635 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:13.931605    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:13.944333 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:13.944310    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:14.098864 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:14.098831    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:14.112224 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:14.112199    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:14.945637 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:14.945608    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:14.958596 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:14.958569    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:15.114681 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:15.114646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:15.127214 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:15.127186    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:15.985637 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:15.985613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:15.998468 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:15.998442    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:16.154812 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:16.154750    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:16.166607 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:16.166577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:16.996957 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:16.996929    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:17.010032 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:17.009998    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:17.172293 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:17.172260    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:17.185179 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:17.185153    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:18.016906 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:18.016878    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:18.029897 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:18.029873    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:18.181132 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:18.181095    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:18.192875 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:18.192855    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:19.080016 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:19.079983    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:19.093616 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:19.093591    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:19.244240 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:19.244208    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:19.257406 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:19.257379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:20.085952 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:20.085912    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:20.103409 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:20.103380    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:20.254002 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:20.253965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:20.266766 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:20.266738    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:21.098042 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:21.098013    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:21.111690 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:21.111665    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:21.267662 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:21.267631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:21.283004 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:21.282973    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:22.145420 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:22.145389    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:22.158433 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:22.158408    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:22.321321 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:22.321287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:22.334250 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:22.334212    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:23.207467 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:23.207429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:23.220295 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:23.220261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:23.375227 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:23.375168    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:23.387672 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:23.387643    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:24.232910 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:24.232869    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:24.246223 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:24.246197    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:24.411373 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:24.411342    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:24.422984 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:24.422955    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:25.262659 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:25.262630    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:25.276369 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:25.276339    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:25.441007 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:25.440967    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:25.456889 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:25.456858    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:26.282353 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:26.282321    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:26.295676 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:26.295655    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:26.457761 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:26.457715    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:26.469732 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:26.469704    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:28.594543 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.594505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6zt68/must-gather-qz5r8"]
Jul 30 00:14:28.595039 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.594914    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerName="main"
Jul 30 00:14:28.595039 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.594927    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerName="main"
Jul 30 00:14:28.595039 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.595010    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7739187e-329c-42b9-a67c-cb30c7bd4cf5" containerName="main"
Jul 30 00:14:28.597319 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.597300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.603187 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.603166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zt68\"/\"kube-root-ca.crt\""
Jul 30 00:14:28.604363 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.604069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6zt68\"/\"default-dockercfg-k9xqk\""
Jul 30 00:14:28.604363 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.604086    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6zt68\"/\"openshift-service-ca.crt\""
Jul 30 00:14:28.613971 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.613944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/must-gather-qz5r8"]
Jul 30 00:14:28.714445 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.714413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slhn\" (UniqueName: \"kubernetes.io/projected/4ea9e8ca-0609-42c0-b5d6-3fa44d22416a-kube-api-access-5slhn\") pod \"must-gather-qz5r8\" (UID: \"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a\") " pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.714633 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.714469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ea9e8ca-0609-42c0-b5d6-3fa44d22416a-must-gather-output\") pod \"must-gather-qz5r8\" (UID: \"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a\") " pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.815919 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.815886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slhn\" (UniqueName: \"kubernetes.io/projected/4ea9e8ca-0609-42c0-b5d6-3fa44d22416a-kube-api-access-5slhn\") pod \"must-gather-qz5r8\" (UID: \"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a\") " pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.816099 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.815954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ea9e8ca-0609-42c0-b5d6-3fa44d22416a-must-gather-output\") pod \"must-gather-qz5r8\" (UID: \"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a\") " pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.816307 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.816287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ea9e8ca-0609-42c0-b5d6-3fa44d22416a-must-gather-output\") pod \"must-gather-qz5r8\" (UID: \"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a\") " pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.823952 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.823932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slhn\" (UniqueName: \"kubernetes.io/projected/4ea9e8ca-0609-42c0-b5d6-3fa44d22416a-kube-api-access-5slhn\") pod \"must-gather-qz5r8\" (UID: \"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a\") " pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:28.906982 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:28.906903    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/must-gather-qz5r8"
Jul 30 00:14:29.052441 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:29.049168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/must-gather-qz5r8"]
Jul 30 00:14:29.054288 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:29.054266    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 00:14:29.980897 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:29.980822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/must-gather-qz5r8" event={"ID":"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a","Type":"ContainerStarted","Data":"52d56353e215996a4776beffc4aabfc886d6c6d997eb7b0af3b7de927eb46662"}
Jul 30 00:14:29.980897 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:29.980876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/must-gather-qz5r8" event={"ID":"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a","Type":"ContainerStarted","Data":"13d85cd2b759fb3dfbf4bb7392bf4e746948d2f6a176fa058ac308805bd17a12"}
Jul 30 00:14:30.993118 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:30.993068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/must-gather-qz5r8" event={"ID":"4ea9e8ca-0609-42c0-b5d6-3fa44d22416a","Type":"ContainerStarted","Data":"da142b7ff7136adae3854b06601717813287a391eb041c1faebe7d540627d47b"}
Jul 30 00:14:31.008750 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:31.008698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6zt68/must-gather-qz5r8" podStartSLOduration=2.195459497 podStartE2EDuration="3.008683324s" totalImagesPullingTime="813.223827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:14:31.007201245 +0000 UTC m=+5286.915891633" watchObservedRunningTime="2026-07-30 00:14:31.008683324 +0000 UTC m=+5286.917373681"
Jul 30 00:14:31.452852 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:31.452823    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x8j5c_13db78f9-f4eb-405b-b632-2a9eed835018/global-pull-secret-syncer/0.log"
Jul 30 00:14:31.542207 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:31.542177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c6vx2_8728adf3-07a6-4869-8405-6e7ba6a7ec6f/konnectivity-agent/0.log"
Jul 30 00:14:31.633638 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:31.633588    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-193.ec2.internal_04c84e703d5815c2c5d803ec17e0971e/haproxy/0.log"
Jul 30 00:14:37.316825 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:37.316774    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-b9spq_9bebaadb-3b3d-476d-8fea-8c83f55be22c/manager/0.log"
Jul 30 00:14:37.341492 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:37.341422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-dccw7_be056854-f42a-407f-bc8e-6bc3b64839d9/manager/0.log"
Jul 30 00:14:37.568305 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:37.568215    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-tpm8g_70dc8007-2ad5-4595-a170-adb08518b5e1/manager/0.log"
Jul 30 00:14:37.594312 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:37.594120    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-rn5v2_ebce846c-4a18-47d1-aa6f-5aca4a7756ba/limitador/0.log"
Jul 30 00:14:39.087876 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:39.087831    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-tg52h_227b4dbc-e960-4432-b261-f7cf67a14aba/cluster-monitoring-operator/0.log"
Jul 30 00:14:39.260525 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:39.260497    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-tg5vj_9d2b5b95-5745-41b2-84f2-92d7b9a3f262/monitoring-plugin/0.log"
Jul 30 00:14:39.295806 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:39.295760    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zb96_cece57cc-fa61-43a5-b75a-cfb4e4285af7/node-exporter/0.log"
Jul 30 00:14:39.316210 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:39.316184    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zb96_cece57cc-fa61-43a5-b75a-cfb4e4285af7/kube-rbac-proxy/0.log"
Jul 30 00:14:39.345898 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:39.345825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zb96_cece57cc-fa61-43a5-b75a-cfb4e4285af7/init-textfile/0.log"
Jul 30 00:14:40.004375 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:40.004338    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/thanos-query/0.log"
Jul 30 00:14:40.026991 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:40.026964    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy-web/0.log"
Jul 30 00:14:40.051042 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:40.051011    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy/0.log"
Jul 30 00:14:40.073437 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:40.073404    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/prom-label-proxy/0.log"
Jul 30 00:14:40.094454 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:40.094420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy-rules/0.log"
Jul 30 00:14:40.115774 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:40.115745    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b66997cdf-r8knr_264210de-4bf3-499e-af03-fd17c0b22984/kube-rbac-proxy-metrics/0.log"
Jul 30 00:14:41.849775 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.849741    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"]
Jul 30 00:14:41.855861 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.855835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.862356 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.862273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"]
Jul 30 00:14:41.862627 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.862575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-proc\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.862741 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.862632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-lib-modules\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.862848 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.862755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mcs\" (UniqueName: \"kubernetes.io/projected/824a273a-6453-4495-b70c-5f1d76ea093b-kube-api-access-s8mcs\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.862848 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.862782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-podres\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.862848 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.862831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-sys\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964163 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mcs\" (UniqueName: \"kubernetes.io/projected/824a273a-6453-4495-b70c-5f1d76ea093b-kube-api-access-s8mcs\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964359 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-podres\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964359 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-sys\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964359 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-proc\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964359 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-lib-modules\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964577 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964445    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-lib-modules\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.964955 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.964920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-podres\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.965086 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.965000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-sys\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.965086 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.965045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/824a273a-6453-4495-b70c-5f1d76ea093b-proc\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:41.975885 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:41.975821    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mcs\" (UniqueName: \"kubernetes.io/projected/824a273a-6453-4495-b70c-5f1d76ea093b-kube-api-access-s8mcs\") pod \"perf-node-gather-daemonset-95z4w\" (UID: \"824a273a-6453-4495-b70c-5f1d76ea093b\") " pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:42.080782 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:42.080749    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54bbdcdd48-chw9c_eddb29a1-f303-49a6-8836-6022d7dc38fa/console/0.log"
Jul 30 00:14:42.172718 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:42.172398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:42.348703 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:42.348667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"]
Jul 30 00:14:43.091665 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.091621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w" event={"ID":"824a273a-6453-4495-b70c-5f1d76ea093b","Type":"ContainerStarted","Data":"9f3b897c6786445de7f6f35a1fd6b76fef830f36012067e24dffce5f6d15e800"}
Jul 30 00:14:43.092192 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.091694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w" event={"ID":"824a273a-6453-4495-b70c-5f1d76ea093b","Type":"ContainerStarted","Data":"9d0301c388019c83337545984a1550cc68d8512bf37762a2593cb779aa3ec9e6"}
Jul 30 00:14:43.092478 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.092452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:43.111032 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.110977    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w" podStartSLOduration=2.110958237 podStartE2EDuration="2.110958237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:14:43.106557263 +0000 UTC m=+5299.015247626" watchObservedRunningTime="2026-07-30 00:14:43.110958237 +0000 UTC m=+5299.019648595"
Jul 30 00:14:43.725182 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.725155    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wt4mt_7c721bef-22cf-40a8-878c-f071e3a1f012/dns/0.log"
Jul 30 00:14:43.744588 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.744559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wt4mt_7c721bef-22cf-40a8-878c-f071e3a1f012/kube-rbac-proxy/0.log"
Jul 30 00:14:43.840040 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:43.840007    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ts59j_df0d9576-7662-40f9-b242-50eb3acada25/dns-node-resolver/0.log"
Jul 30 00:14:44.384601 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:44.384570    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29756160-sjnbv_1f9d8301-e499-4bde-ba27-e89560ea09bf/image-pruner/0.log"
Jul 30 00:14:44.482962 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:44.482903    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hg8hl_062d05c7-b02b-464f-abb6-3c6f113ee0ce/node-ca/0.log"
Jul 30 00:14:45.696467 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:45.696416    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:14:45.717979 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:45.717948    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:14:45.748683 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:45.748653    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:14:45.769192 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:45.769161    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:14:46.266140 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:46.266059    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4hpqx_00a5e172-5fa6-4f97-b08b-33d512358027/serve-healthcheck-canary/0.log"
Jul 30 00:14:46.778101 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:46.778067    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-dd689_258f5c2b-06f6-4659-a589-afdbe1c6bf5e/insights-operator/0.log"
Jul 30 00:14:46.784259 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:46.784231    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-dd689_258f5c2b-06f6-4659-a589-afdbe1c6bf5e/insights-operator/1.log"
Jul 30 00:14:46.880426 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:46.880392    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rxsh_0775c5fc-3a94-46dd-8691-b29c2db1e186/kube-rbac-proxy/0.log"
Jul 30 00:14:46.901179 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:46.901148    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rxsh_0775c5fc-3a94-46dd-8691-b29c2db1e186/exporter/0.log"
Jul 30 00:14:46.921412 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:46.921374    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rxsh_0775c5fc-3a94-46dd-8691-b29c2db1e186/extractor/0.log"
Jul 30 00:14:49.111434 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:49.111406    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6zt68/perf-node-gather-daemonset-95z4w"
Jul 30 00:14:50.615846 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:50.615816    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-844cf74686-w286v_ced5244f-c645-46e2-ab45-38b672cab11b/manager/0.log"
Jul 30 00:14:51.276014 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:51.275979    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qvr5s_47088f3b-a2d1-4d78-bb79-7c969fdaeb65/seaweedfs/0.log"
Jul 30 00:14:51.942710 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:51.942604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:14:51.957491 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:51.957453    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:14:51.988809 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:51.988102    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:14:52.006516 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:52.006480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:14:53.251535 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:53.251501    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dz27l_bbd945e8-9126-4f9a-939b-2b2844bb8443/discovery/0.log"
Jul 30 00:14:53.280047 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:53.280010    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/0.log"
Jul 30 00:14:53.315150 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:53.315116    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-zrlbw_6e724f3c-88f2-4fcc-924e-95054f341fd2/istio-proxy/1.log"
Jul 30 00:14:53.334664 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:53.334633    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d7fb7b5bf-sg26j_64791bc3-b8dd-4974-bd07-96d0bed3c998/router/0.log"
Jul 30 00:14:57.775285 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.775256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/kube-multus-additional-cni-plugins/0.log"
Jul 30 00:14:57.798155 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.798123    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/egress-router-binary-copy/0.log"
Jul 30 00:14:57.818020 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.817995    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/cni-plugins/0.log"
Jul 30 00:14:57.837723 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.837695    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/bond-cni-plugin/0.log"
Jul 30 00:14:57.857761 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.857731    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/routeoverride-cni/0.log"
Jul 30 00:14:57.877338 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.877307    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/whereabouts-cni-bincopy/0.log"
Jul 30 00:14:57.896034 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:57.896005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bgxrn_e01dc35e-2818-4861-913c-f4df8df8f334/whereabouts-cni/0.log"
Jul 30 00:14:58.316701 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:58.316665    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p94q8_95ffbc47-f716-43e4-96f4-7f4f97ee47ed/kube-multus/0.log"
Jul 30 00:14:58.496711 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:58.496680    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vct9g_af595ab0-8515-4923-af82-6cdcd20b5c57/network-metrics-daemon/0.log"
Jul 30 00:14:58.533607 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:58.533579    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vct9g_af595ab0-8515-4923-af82-6cdcd20b5c57/kube-rbac-proxy/0.log"
Jul 30 00:14:59.338291 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.338261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/ovn-controller/0.log"
Jul 30 00:14:59.353485 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.353461    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/ovn-acl-logging/0.log"
Jul 30 00:14:59.376415 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.376391    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/kube-rbac-proxy-node/0.log"
Jul 30 00:14:59.396004 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.395982    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 00:14:59.410557 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.410503    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/northd/0.log"
Jul 30 00:14:59.428761 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.428744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/nbdb/0.log"
Jul 30 00:14:59.449848 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.449822    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/sbdb/0.log"
Jul 30 00:14:59.510070 ip-10-0-131-193 kubenswrapper[2580]: I0730 00:14:59.510049    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grsjn_ba52014f-ffe4-4963-b240-5d8271d599a0/ovnkube-controller/0.log"