Jul 01 18:35:06.767363 ip-10-0-133-57 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 18:35:06.767378 ip-10-0-133-57 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 18:35:06.767388 ip-10-0-133-57 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 18:35:06.767703 ip-10-0-133-57 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 18:35:16.982890 ip-10-0-133-57 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 18:35:16.982906 ip-10-0-133-57 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9c65ea290b5d4773acc8d54254a6e281 --
Jul 01 18:37:20.478700 ip-10-0-133-57 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 18:37:22.050884 ip-10-0-133-57 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:22.050884 ip-10-0-133-57 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 18:37:22.050884 ip-10-0-133-57 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:22.050884 ip-10-0-133-57 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 18:37:22.050884 ip-10-0-133-57 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 18:37:22.051514 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.050958    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 18:37:22.056389 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056372    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:22.056389 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056388    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:22.056389 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056392    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056396    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056399    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056403    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056405    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056408    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056411    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056414    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056416    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056419    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056421    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056424    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056427    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056430    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056432    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056435    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056437    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056440    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056442    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056445    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:22.056512 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056447    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056450    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056452    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056455    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056457    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056460    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056462    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056465    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056471    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056473    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056476    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056478    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056483    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056487    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056491    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056494    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056499    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056503    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056506    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:22.057124 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056509    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056511    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056514    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056517    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056519    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056522    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056524    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056527    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056531    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056534    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056537    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056539    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056542    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056545    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056548    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056550    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056553    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056555    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056558    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056560    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:22.057581 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056563    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056565    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056567    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056570    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056573    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056575    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056578    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056582    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056584    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056587    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056592    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056595    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056597    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056600    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056602    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056605    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056608    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056610    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056613    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056616    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:22.058067 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056619    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056621    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056624    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056627    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056629    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056631    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056634    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056636    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056639    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056641    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056644    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056646    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056649    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056652    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056654    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056657    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056659    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.056662    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057382    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057389    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:22.058562 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057392    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057395    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057397    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057400    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057402    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057405    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057408    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057410    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057413    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057415    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057418    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057420    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057423    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057425    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057428    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057431    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057433    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057436    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057438    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057441    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:22.059060 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057443    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057446    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057449    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057452    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057454    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057457    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057459    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057462    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057464    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057468    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057470    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057473    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057476    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057478    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057481    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057485    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057489    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057492    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057496    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:22.059566 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057499    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057502    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057504    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057507    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057509    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057512    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057514    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057518    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057521    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057524    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057527    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057530    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057533    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057535    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057538    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057544    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057547    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057549    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057552    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057554    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:22.060051 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057556    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057559    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057562    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057564    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057567    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057570    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057572    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057575    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057577    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057580    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057582    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057585    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057587    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057590    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057593    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057596    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057598    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057601    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057603    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057606    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:22.060533 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057608    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057611    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057617    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057620    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057622    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057625    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057628    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057632    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057635    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057637    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057640    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057643    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057647    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057650    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057653    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057655    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057658    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.057661    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058050    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:22.061028 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058056    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058058    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058061    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058064    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058066    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058069    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058072    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058074    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058077    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058080    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058082    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058085    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058087    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058090    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058092    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058096    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058098    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058101    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058103    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058106    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:22.061484 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058110    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058113    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058115    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058118    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058120    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058123    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058125    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058128    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058131    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058133    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058136    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058138    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058141    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058143    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058146    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058148    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058151    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058154    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058156    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058158    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:22.061953 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058161    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058163    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058166    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058168    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058171    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058173    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058176    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058178    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058181    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058184    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058186    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058188    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058192    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058195    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058198    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058201    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058204    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058207    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058209    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:22.062428 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058214    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058219    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058222    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058225    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058228    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058231    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058234    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058237    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058239    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058243    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058246    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058249    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058251    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058253    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058256    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058258    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058261    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058263    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058265    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:22.062893 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058268    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058270    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058273    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058276    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058278    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058280    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058285    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058289    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058292    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058294    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058299    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058302    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058304    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058307    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058309    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058312    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058314    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058317    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.058320    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:22.063345 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058386    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058393    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058399    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058403    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058408    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058411    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058415    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058420    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058423    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058427    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058431    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058434    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058437    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058440    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058443    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058446    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058450    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058452    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058458    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058461    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058464    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058467    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058471    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058474    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 18:37:22.063860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058477    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058481    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058484    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058487    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058490    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058493    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058496    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058499    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058503    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058507    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058510    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058513    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058515    2577 flags.go:64] FLAG: --enable-server="true"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058518    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058522    2577 flags.go:64] FLAG: --event-burst="100"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058525    2577 flags.go:64] FLAG: --event-qps="50"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058528    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058532    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058535    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058539    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058542    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058545    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058548    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058551    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058554    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 18:37:22.064422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058557    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058562    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058565    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058568    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058571    2577 flags.go:64] FLAG: --feature-gates=""
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058575    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058578    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058581    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058585    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058588    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058590    2577 flags.go:64] FLAG: --help="false"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058593    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058596    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058599    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058602    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058606    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058609    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058612    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058615    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058618    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058621    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058624    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058627    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058630    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 01 18:37:22.065070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058633    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058636    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058639    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058642    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058645    2577 flags.go:64] FLAG: --lock-file=""
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058647    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058650    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058653    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058658    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058663    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058666    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058668    2577 flags.go:64] FLAG: --logging-format="text"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058671    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058675    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058678    2577 flags.go:64] FLAG: --manifest-url=""
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058680    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058690    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058693    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058698    2577 flags.go:64] FLAG: --max-pods="110"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058701    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058704    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058706    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058709    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058712    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058715    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 18:37:22.065668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058718    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058725    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058728    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058731    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058734    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058737    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058742    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058746    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058750    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058753    2577 flags.go:64] FLAG: --port="10250"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058756    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058759    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bf3a8c34b2931334"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058762    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058766    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058769    2577 flags.go:64] FLAG: --register-node="true"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058771    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058775    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058778    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058781    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058784    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058787    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058791    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058794    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058796    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058799    2577 flags.go:64] FLAG: --runonce="false"
Jul 01 18:37:22.066283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058815    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058818    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058822    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058825    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058828    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058831    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058834    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058837    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058839    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058842    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058845    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058848    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058851    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058854    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058859    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058864    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058867    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058870    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058876    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058879    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058882    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058885    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058888    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058891    2577 flags.go:64] FLAG: --v="2"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058896    2577 flags.go:64] FLAG: --version="false"
Jul 01 18:37:22.066877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058901    2577 flags.go:64] FLAG: --vmodule=""
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058904    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.058907    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059027    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059031    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059034    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059037    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059039    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059042    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059044    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059047    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059050    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059052    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059055    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059057    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059060    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059063    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059065    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059068    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059070    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:22.067471 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059073    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059077    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059079    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059082    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059084    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059087    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059089    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059092    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059095    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059097    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059100    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059102    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059105    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059108    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059111    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059113    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059116    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059118    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059120    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059123    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:22.067977 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059125    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059128    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059130    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059133    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059135    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059138    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059140    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059143    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059145    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059149    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059153    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059156    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059158    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059162    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059165    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059167    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059170    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059172    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059176    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059178    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:22.068479 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059181    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059184    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059186    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059189    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059191    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059194    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059197    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059199    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059203    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059207    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059210    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059213    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059215    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059218    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059221    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059224    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059226    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059229    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059231    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059234    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:22.068980 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059236    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059239    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059241    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059244    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059247    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059251    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059253    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059256    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059258    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059261    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059264    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059267    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059270    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059274    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059277    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059280    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059283    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059285    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059288    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:22.069456 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059291    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059293    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.059296    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.060108    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.066937    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.066951    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.067744    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.067754    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067823    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067828    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067831    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067834    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067838    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067840    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067844    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:22.069920 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067846    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067849    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067852    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067854    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067857    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067859    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067862    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067865    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067867    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067869    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067873    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067875    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067878    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067881    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067883    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067886    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067889    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067891    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067894    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:22.070280 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067896    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067899    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067901    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067905    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067908    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067911    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067913    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067916    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067919    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067921    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067924    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067927    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067930    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067932    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067935    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067937    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067940    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067942    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067945    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067947    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:22.070740 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067950    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067952    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067955    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067958    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067963    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067966    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067969    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067972    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067974    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067977    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067979    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067983    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067986    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067989    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067992    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067994    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.067997    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068000    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068002    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068005    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:22.071288 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068007    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068010    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068013    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068015    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068019    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068023    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068026    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068029    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068033    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068037    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068040    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068043    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068045    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068048    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068051    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068054    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068056    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068059    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068062    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:22.071765 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068065    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068067    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068070    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068072    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068075    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068078    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068080    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068083    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068085    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068088    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068090    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068093    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068095    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068098    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.068103    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:22.072274 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068228    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068232    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068235    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068238    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068241    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068243    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068246    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068249    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068252    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068254    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068257    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068259    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068262    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068266    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068269    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068272    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068276    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068280    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068283    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 18:37:22.072639 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068285    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068288    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068290    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068293    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068295    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068298    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068300    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068302    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068305    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068307    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068310    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068312    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068314    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068317    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068319    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068322    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068324    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068327    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068329    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068332    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 18:37:22.073123 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068334    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068337    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068339    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068342    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068344    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068347    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068349    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068352    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068354    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068357    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068360    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068363    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068365    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068367    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068370    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068372    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068375    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068377    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068380    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068382    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 18:37:22.073600 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068385    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068387    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068390    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068392    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068395    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068397    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068400    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068402    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068406    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068409    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068412    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068415    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068418    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068420    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068423    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068425    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068428    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068430    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068433    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068436    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 18:37:22.074086 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068438    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068441    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068443    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068446    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068448    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068451    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068453    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068456    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068458    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068461    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068463    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068465    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068468    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068470    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068473    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068475    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068478    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068481    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068483    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 18:37:22.074555 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:22.068486    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 18:37:22.075016 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.068491    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 18:37:22.075016 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.068595    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 18:37:22.075016 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.070781    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 18:37:22.075016 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.072784    2577 server.go:1019] "Starting client certificate rotation"
Jul 01 18:37:22.075016 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.072834    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:37:22.075016 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.072864    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 18:37:22.076464 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.076011    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:37:22.078669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.078651    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 18:37:22.087444 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.087423    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 01 18:37:22.089568 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.089555    2577 log.go:25] "Validated CRI v1 image API"
Jul 01 18:37:22.090421 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.090401    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 18:37:22.095221 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.095203    2577 fs.go:135] Filesystem UUIDs: map[16cd2b7e-f382-4e53-a3ac-a2ce683307ee:/dev/nvme0n1p4 471a9c95-d57f-4b75-b543-00402470570d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 01 18:37:22.095280 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.095221    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 18:37:22.101387 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.101278    2577 manager.go:216] Machine: {Timestamp:2026-07-01 18:37:22.098995021 +0000 UTC m=+0.333949298 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099694 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f10b086d612c85759b96a3e661d2 SystemUUID:ec27f10b-086d-612c-8575-9b96a3e661d2 BootID:9c65ea29-0b5d-4773-acc8-d54254a6e281 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:be:57:49:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:be:57:49:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:30:aa:4e:05:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 18:37:22.101387 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.101381    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 18:37:22.101509 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.101498    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 18:37:22.104570 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.104538    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 18:37:22.104720 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.104573    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 18:37:22.105265 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.105256    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 18:37:22.105303 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.105269    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 18:37:22.105303 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.105277    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:37:22.105875 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.105865    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 18:37:22.106756 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.106746    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 18:37:22.108165 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.108148    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:37:22.109777 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109765    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:37:22.109904 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109892    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 18:37:22.109952 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109922    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 18:37:22.109952 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109931    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 18:37:22.109952 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109945    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 18:37:22.110049 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109954    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 01 18:37:22.110049 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.109965    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 18:37:22.110432 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.110420    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:37:22.110479 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.110438    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 18:37:22.112608 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.112576    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 18:37:22.114915 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.114890    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 18:37:22.115008 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.114946    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 18:37:22.115678 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115654    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 18:37:22.115678 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115673    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 18:37:22.115678 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115681    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115687    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115697    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115707    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115719    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115774    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115792    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115821    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115836    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 18:37:22.115864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.115852    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 18:37:22.116754 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.116739    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 18:37:22.116827 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.116757    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 18:37:22.120343 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.120326    2577 server.go:1268] "Started kubelet"
Jul 01 18:37:22.120528 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.120478    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 18:37:22.121117 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.120823    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 18:37:22.121117 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.120901    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 18:37:22.121208 ip-10-0-133-57 systemd[1]: Started Kubernetes Kubelet.
Jul 01 18:37:22.122625 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.122609    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 18:37:22.123313 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.123283    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 18:37:22.123967 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.123945    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 18:37:22.123967 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.123947    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 18:37:22.124062 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.124055    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 18:37:22.127366 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.127346    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 18:37:22.130196 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130177    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 18:37:22.130196 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130190    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 18:37:22.130897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130745    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 18:37:22.130897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130759    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 18:37:22.130897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130773    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 18:37:22.131057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130951    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 18:37:22.131057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.130962    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 18:37:22.131141 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.130287    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.131292 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131278    2577 factory.go:55] Registering systemd factory
Jul 01 18:37:22.131292 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131293    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 01 18:37:22.131494 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131481    2577 factory.go:153] Registering CRI-O factory
Jul 01 18:37:22.131494 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131494    2577 factory.go:223] Registration of the crio container factory successfully
Jul 01 18:37:22.131583 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131566    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 18:37:22.131618 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131585    2577 factory.go:103] Registering Raw factory
Jul 01 18:37:22.131618 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.131596    2577 manager.go:1196] Started watching for new ooms in manager
Jul 01 18:37:22.132243 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.132063    2577 manager.go:349] Starting recovery of all containers
Jul 01 18:37:22.134830 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.134786    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 18:37:22.134915 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.134900    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 18:37:22.135911 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.134988    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-57.ec2.internal.18be3e562098ecd7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-57.ec2.internal,UID:ip-10-0-133-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-57.ec2.internal,},FirstTimestamp:2026-07-01 18:37:22.120297687 +0000 UTC m=+0.355251946,LastTimestamp:2026-07-01 18:37:22.120297687 +0000 UTC m=+0.355251946,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-57.ec2.internal,}"
Jul 01 18:37:22.143158 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.143132    2577 manager.go:354] Recovery completed
Jul 01 18:37:22.148745 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.148730    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.150486 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.150469    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.150581 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.150505    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.150581 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.150520    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.151094 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151078    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 18:37:22.151094 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151092    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 18:37:22.151194 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151107    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 18:37:22.151194 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151085    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.151763 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151750    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.151837 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151776    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.151837 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.151787    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.152483 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.152472    2577 policy_none.go:49] "None policy: Start"
Jul 01 18:37:22.152521 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.152487    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 18:37:22.152521 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.152497    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 18:37:22.153360 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.153297    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-57.ec2.internal.18be3e5622659730  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-57.ec2.internal,UID:ip-10-0-133-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-57.ec2.internal,},FirstTimestamp:2026-07-01 18:37:22.150487856 +0000 UTC m=+0.385442119,LastTimestamp:2026-07-01 18:37:22.150487856 +0000 UTC m=+0.385442119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-57.ec2.internal,}"
Jul 01 18:37:22.153596 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.153585    2577 policy_none.go:47] "Start"
Jul 01 18:37:22.163584 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.163566    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9v6v2"
Jul 01 18:37:22.165856 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.165759    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-57.ec2.internal.18be3e562265f413  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-57.ec2.internal,UID:ip-10-0-133-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-57.ec2.internal,},FirstTimestamp:2026-07-01 18:37:22.150511635 +0000 UTC m=+0.385465893,LastTimestamp:2026-07-01 18:37:22.150511635 +0000 UTC m=+0.385465893,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-57.ec2.internal,}"
Jul 01 18:37:22.175541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.175518    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9v6v2"
Jul 01 18:37:22.175966 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.175890    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-57.ec2.internal.18be3e5622662c0d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-57.ec2.internal,UID:ip-10-0-133-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-57.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-57.ec2.internal,},FirstTimestamp:2026-07-01 18:37:22.150525965 +0000 UTC m=+0.385480228,LastTimestamp:2026-07-01 18:37:22.150525965 +0000 UTC m=+0.385480228,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-57.ec2.internal,}"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.192862    2577 manager.go:337] "Starting Device Plugin manager"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.192889    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.192900    2577 server.go:87] "Starting device plugin registration server"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.193144    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.193157    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.193262    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.193362    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.193382    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.193835    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.193895    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.200523    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.201651    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.201698    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.201724    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.201763    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 18:37:22.207537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.204205    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:22.294276 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.294229    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.295293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.295271    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.295397 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.295305    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.295397 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.295316    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.295397 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.295347    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.302435 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.302417    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"]
Jul 01 18:37:22.302506 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.302490    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.303283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.303258    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.303283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.303287    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.303403 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.303296    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.304609 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.304597    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.304751 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.304740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.304789 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.304766    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.305297 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.305282    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.305377 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.305294    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.305377 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.305306    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.305377 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.305318    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.305465 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.305318    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.305495 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.305467    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.306766 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.306695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.306766 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.306726    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 18:37:22.307341 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.307325    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 18:37:22.307425 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.307351    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 18:37:22.307425 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.307366    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeHasSufficientPID"
Jul 01 18:37:22.307842 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.307830    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.307885 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.307848    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-57.ec2.internal\": node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.332912 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.332892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c582d8238a6f3daead6044036f2a5eeb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-57.ec2.internal\" (UID: \"c582d8238a6f3daead6044036f2a5eeb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.333012 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.332921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/469f128e84139c59dc87d077aeeac40f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal\" (UID: \"469f128e84139c59dc87d077aeeac40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.333012 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.332942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/469f128e84139c59dc87d077aeeac40f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal\" (UID: \"469f128e84139c59dc87d077aeeac40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.339496 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.339477    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-57.ec2.internal\" not found" node="ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.343824 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.343797    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-57.ec2.internal\" not found" node="ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.349288 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.349270    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.433536 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.433506    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/469f128e84139c59dc87d077aeeac40f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal\" (UID: \"469f128e84139c59dc87d077aeeac40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.433536 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.433473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/469f128e84139c59dc87d077aeeac40f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal\" (UID: \"469f128e84139c59dc87d077aeeac40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.433713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.433578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/469f128e84139c59dc87d077aeeac40f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal\" (UID: \"469f128e84139c59dc87d077aeeac40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.433713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.433594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c582d8238a6f3daead6044036f2a5eeb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-57.ec2.internal\" (UID: \"c582d8238a6f3daead6044036f2a5eeb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.433713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.433621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c582d8238a6f3daead6044036f2a5eeb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-57.ec2.internal\" (UID: \"c582d8238a6f3daead6044036f2a5eeb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.433713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.433646    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/469f128e84139c59dc87d077aeeac40f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal\" (UID: \"469f128e84139c59dc87d077aeeac40f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.450316 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.450294    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.551374 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.551347    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.643493 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.643431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.648393 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:22.648376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:22.651427 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.651411    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.751885 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.751846    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.852356 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.852325    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:22.952941 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:22.952863    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:23.053338 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:23.053307    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:23.072482 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.072459    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 18:37:23.072609 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.072593    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 18:37:23.142144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.142118    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:23.153655 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:23.153509    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-57.ec2.internal\" not found"
Jul 01 18:37:23.162044 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.162024    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:23.180127 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.180089    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 18:32:22 +0000 UTC" deadline="2028-03-21 13:01:42.2890518 +0000 UTC"
Jul 01 18:37:23.180250 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.180155    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15090h24m19.108907241s"
Jul 01 18:37:23.230650 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.230588    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:23.246500 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.246471    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:37:23.247643 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.247626    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal"
Jul 01 18:37:23.260411 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.260389    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 18:37:23.261397 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:23.261370    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469f128e84139c59dc87d077aeeac40f.slice/crio-3af87bebb4b5fc53363bc48f4bc161815928d2006e2d153596ec67a7e7c60577 WatchSource:0}: Error finding container 3af87bebb4b5fc53363bc48f4bc161815928d2006e2d153596ec67a7e7c60577: Status 404 returned error can't find the container with id 3af87bebb4b5fc53363bc48f4bc161815928d2006e2d153596ec67a7e7c60577
Jul 01 18:37:23.261711 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:23.261692    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc582d8238a6f3daead6044036f2a5eeb.slice/crio-b0b2e25556f7d78e0bcdda44cea0f32740b73aac1a8a381ad2d09ad2b8350e6a WatchSource:0}: Error finding container b0b2e25556f7d78e0bcdda44cea0f32740b73aac1a8a381ad2d09ad2b8350e6a: Status 404 returned error can't find the container with id b0b2e25556f7d78e0bcdda44cea0f32740b73aac1a8a381ad2d09ad2b8350e6a
Jul 01 18:37:23.267497 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.267476    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:37:23.561217 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.561148    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:23.936607 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:23.936529    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 18:37:24.111442 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.111411    2577 apiserver.go:52] "Watching apiserver"
Jul 01 18:37:24.118387 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.118360    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 18:37:24.121647 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.121614    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-29kff","kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q","openshift-cluster-node-tuning-operator/tuned-w692r","openshift-dns/node-resolver-v6j5f","openshift-image-registry/node-ca-k5qgm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal","openshift-multus/multus-additional-cni-plugins-smlvd","kube-system/global-pull-secret-syncer-k499p","openshift-multus/multus-wqlzd","openshift-multus/network-metrics-daemon-xkqt7","openshift-network-diagnostics/network-check-target-c69kt","openshift-network-operator/iptables-alerter-9wmgw","openshift-ovn-kubernetes/ovnkube-node-n4bhn"]
Jul 01 18:37:24.123486 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.123361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.124782 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.124608    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.125923 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.125901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.127314 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.127295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.128390 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.128370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.129759 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.129739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.131188 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.131171    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 18:37:24.131584 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.131553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.131660 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.131619    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:24.132856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.132836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.134378 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.134356    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.134878 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.134848    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.135114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.135097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.135363 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.135344    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c2qf7\""
Jul 01 18:37:24.136075 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.136057    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 18:37:24.136164 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.136097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 18:37:24.136305 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.136287    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.136479 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.136460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.137490 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.137399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.137677 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.137659    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5zrwh\""
Jul 01 18:37:24.137742 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.137721    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 18:37:24.138505 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.138170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q9b5h\""
Jul 01 18:37:24.138505 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.138451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.139560 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.138872    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.139560 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.139214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.139560 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.139485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.140825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.140136    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vtj9v\""
Jul 01 18:37:24.140825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.140261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xksl\""
Jul 01 18:37:24.140825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.140716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.141000 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.140850    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:24.142046 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9ce6483-0300-475e-855c-e5237856a12f-multus-daemon-config\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.142163 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-etc-kubernetes\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.142225 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.142225 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvxdn\" (UniqueName: \"kubernetes.io/projected/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-kube-api-access-xvxdn\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.142309 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28885980-38ee-46ba-9944-e7ce6739496e-iptables-alerter-script\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.142309 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-k8s-cni-cncf-io\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.142407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-sys-fs\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.142407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:24.142407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysconfig\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.142407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cba52558-8c44-4d21-94cf-ed28570c6337-kubelet-config\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.142407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.142407 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.142402    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:24.142667 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28885980-38ee-46ba-9944-e7ce6739496e-host-slash\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.142667 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-cnibin\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.142667 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9ce6483-0300-475e-855c-e5237856a12f-cni-binary-copy\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.142667 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-netns\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.142830 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-socket-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.142830 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysctl-d\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.142830 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb73641-462d-4970-98c5-d491835f923f-serviceca\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.142955 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.142831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-hostroot\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143174 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-multus-certs\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143229 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-device-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.143229 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-etc-selinux\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.143324 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-systemd\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143324 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-sys\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143324 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-cni-multus\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143324 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysctl-conf\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143324 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7qp\" (UniqueName: \"kubernetes.io/projected/8aca6411-d820-4c64-bec2-2ef05551c627-kube-api-access-zr7qp\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143552 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cba52558-8c44-4d21-94cf-ed28570c6337-dbus\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.143552 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6sb\" (UniqueName: \"kubernetes.io/projected/28885980-38ee-46ba-9944-e7ce6739496e-kube-api-access-8t6sb\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.143552 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-cni-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143552 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-os-release\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143552 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-socket-dir-parent\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143552 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-conf-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-modprobe-d\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-run\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-var-lib-kubelet\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-host\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-cni-bin\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-kubelet\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45frk\" (UniqueName: \"kubernetes.io/projected/c9ce6483-0300-475e-855c-e5237856a12f-kube-api-access-45frk\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-registration-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnjt\" (UniqueName: \"kubernetes.io/projected/00aedd5e-9e98-414c-930d-9d6661e4a0de-kube-api-access-lbnjt\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-kubernetes\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.143856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8aca6411-d820-4c64-bec2-2ef05551c627-tmp\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-hosts-file\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb73641-462d-4970-98c5-d491835f923f-host\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpms\" (UniqueName: \"kubernetes.io/projected/9cb73641-462d-4970-98c5-d491835f923f-kube-api-access-2tpms\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-lib-modules\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.143989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8aca6411-d820-4c64-bec2-2ef05551c627-etc-tuned\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.144023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-tmp-dir\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.144346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.144045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-system-cni-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.146058 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.145905    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 18:37:24.146058 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.145924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.146058 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.146054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 18:37:24.146205 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.146127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 18:37:24.146244 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.146232    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 18:37:24.146294 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.146281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 18:37:24.146501 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.146485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.146983 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.146963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 18:37:24.147078 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-njrbf\""
Jul 01 18:37:24.147078 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147056    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 18:37:24.147187 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147087    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.147187 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147058    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-46wvf\""
Jul 01 18:37:24.147279 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.147279 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147273    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 18:37:24.147378 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 18:37:24.147378 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147273    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 18:37:24.147378 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147320    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 18:37:24.147378 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.147371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6zd42\""
Jul 01 18:37:24.151218 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.151200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mgcb\""
Jul 01 18:37:24.153218 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.153203    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 18:37:24.202843 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.202751    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 18:37:24.206510 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.206467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal" event={"ID":"469f128e84139c59dc87d077aeeac40f","Type":"ContainerStarted","Data":"3af87bebb4b5fc53363bc48f4bc161815928d2006e2d153596ec67a7e7c60577"}
Jul 01 18:37:24.208310 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.208284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal" event={"ID":"c582d8238a6f3daead6044036f2a5eeb","Type":"ContainerStarted","Data":"b0b2e25556f7d78e0bcdda44cea0f32740b73aac1a8a381ad2d09ad2b8350e6a"}
Jul 01 18:37:24.218472 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.218451    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdvwc"
Jul 01 18:37:24.232386 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.232368    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 18:37:24.244616 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-systemd\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.244717 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-netns\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.244717 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-socket-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.244717 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysctl-d\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.244717 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244718    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-netns\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-cni-netd\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pfs\" (UniqueName: \"kubernetes.io/projected/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-kube-api-access-d7pfs\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb73641-462d-4970-98c5-d491835f923f-serviceca\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244823    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysctl-d\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244824    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-socket-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-hostroot\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244868    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-hostroot\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-multus-certs\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.244921 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovnkube-config\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-env-overrides\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244944    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-multus-certs\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-os-release\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.244985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvcw\" (UniqueName: \"kubernetes.io/projected/9f6cb189-4786-4e41-bd91-0888314a0b65-kube-api-access-prvcw\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-systemd-units\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-etc-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-tuning-conf-dir\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9106ec84-55be-4d4e-be47-97aca4c61dc4-konnectivity-ca\") pod \"konnectivity-agent-29kff\" (UID: \"9106ec84-55be-4d4e-be47-97aca4c61dc4\") " pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-cni-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-os-release\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-socket-dir-parent\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245250    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-os-release\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245267    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cb73641-462d-4970-98c5-d491835f923f-serviceca\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-conf-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245361    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-cni-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.245396 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-run\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-conf-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245364    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-multus-socket-dir-parent\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-var-lib-kubelet\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-host\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245461    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-run\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-host\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-var-lib-kubelet\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-ovn\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-kubelet\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45frk\" (UniqueName: \"kubernetes.io/projected/c9ce6483-0300-475e-855c-e5237856a12f-kube-api-access-45frk\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-registration-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-kubernetes\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245646    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-kubelet\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8aca6411-d820-4c64-bec2-2ef05551c627-tmp\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-tmp-dir\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-system-cni-dir\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-registration-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.246132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpms\" (UniqueName: \"kubernetes.io/projected/9cb73641-462d-4970-98c5-d491835f923f-kube-api-access-2tpms\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8aca6411-d820-4c64-bec2-2ef05551c627-etc-tuned\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-kubernetes\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-kubelet\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-system-cni-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9ce6483-0300-475e-855c-e5237856a12f-multus-daemon-config\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-etc-kubernetes\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245943    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-system-cni-dir\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9106ec84-55be-4d4e-be47-97aca4c61dc4-agent-certs\") pod \"konnectivity-agent-29kff\" (UID: \"9106ec84-55be-4d4e-be47-97aca4c61dc4\") " pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysconfig\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245996    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-etc-kubernetes\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cba52558-8c44-4d21-94cf-ed28570c6337-kubelet-config\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.245978    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246027    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-tmp-dir\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246040    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.247043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysconfig\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.246124    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-node-log\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246138    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cba52558-8c44-4d21-94cf-ed28570c6337-kubelet-config\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-log-socket\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.246191    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:24.746171774 +0000 UTC m=+2.981126027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-cni-bin\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cnibin\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-cnibin\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246355    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-cnibin\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246397    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9ce6483-0300-475e-855c-e5237856a12f-multus-daemon-config\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9ce6483-0300-475e-855c-e5237856a12f-cni-binary-copy\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-device-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-etc-selinux\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-systemd\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-sys\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cba52558-8c44-4d21-94cf-ed28570c6337-dbus\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.247877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6sb\" (UniqueName: \"kubernetes.io/projected/28885980-38ee-46ba-9944-e7ce6739496e-kube-api-access-8t6sb\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.246994    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-device-dir\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-systemd\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247073    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-sys\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-slash\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cba52558-8c44-4d21-94cf-ed28570c6337-dbus\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-cni-multus\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247188    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-cni-multus\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247265    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9ce6483-0300-475e-855c-e5237856a12f-cni-binary-copy\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-etc-selinux\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysctl-conf\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7qp\" (UniqueName: \"kubernetes.io/projected/8aca6411-d820-4c64-bec2-2ef05551c627-kube-api-access-zr7qp\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-run-netns\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-modprobe-d\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28885980-38ee-46ba-9944-e7ce6739496e-iptables-alerter-script\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.248532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247436    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-sysctl-conf\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-cni-bin\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnjt\" (UniqueName: \"kubernetes.io/projected/00aedd5e-9e98-414c-930d-9d6661e4a0de-kube-api-access-lbnjt\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-hosts-file\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-var-lib-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmwp\" (UniqueName: \"kubernetes.io/projected/0aa73a80-6429-4a23-8445-6b1110dddd5b-kube-api-access-tdmwp\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-var-lib-cni-bin\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb73641-462d-4970-98c5-d491835f923f-host\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247677    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-etc-modprobe-d\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247688    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-hosts-file\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-lib-modules\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovn-node-metrics-cert\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8aca6411-d820-4c64-bec2-2ef05551c627-lib-modules\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvxdn\" (UniqueName: \"kubernetes.io/projected/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-kube-api-access-xvxdn\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovnkube-script-lib\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.249145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247830    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cb73641-462d-4970-98c5-d491835f923f-host\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cni-binary-copy\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-k8s-cni-cncf-io\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-sys-fs\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28885980-38ee-46ba-9944-e7ce6739496e-host-slash\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9ce6483-0300-475e-855c-e5237856a12f-host-run-k8s-cni-cncf-io\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.247959    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28885980-38ee-46ba-9944-e7ce6739496e-iptables-alerter-script\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.248026    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28885980-38ee-46ba-9944-e7ce6739496e-host-slash\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.248029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00aedd5e-9e98-414c-930d-9d6661e4a0de-sys-fs\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.249687    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8aca6411-d820-4c64-bec2-2ef05551c627-tmp\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.249894 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.249713    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8aca6411-d820-4c64-bec2-2ef05551c627-etc-tuned\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.268089 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.268070    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdvwc"
Jul 01 18:37:24.304390 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.304362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpms\" (UniqueName: \"kubernetes.io/projected/9cb73641-462d-4970-98c5-d491835f923f-kube-api-access-2tpms\") pod \"node-ca-k5qgm\" (UID: \"9cb73641-462d-4970-98c5-d491835f923f\") " pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.305018 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.304998    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnjt\" (UniqueName: \"kubernetes.io/projected/00aedd5e-9e98-414c-930d-9d6661e4a0de-kube-api-access-lbnjt\") pod \"aws-ebs-csi-driver-node-86f7q\" (UID: \"00aedd5e-9e98-414c-930d-9d6661e4a0de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.305512 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.305493    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvxdn\" (UniqueName: \"kubernetes.io/projected/1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d-kube-api-access-xvxdn\") pod \"node-resolver-v6j5f\" (UID: \"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d\") " pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.319976 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.319956    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45frk\" (UniqueName: \"kubernetes.io/projected/c9ce6483-0300-475e-855c-e5237856a12f-kube-api-access-45frk\") pod \"multus-wqlzd\" (UID: \"c9ce6483-0300-475e-855c-e5237856a12f\") " pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.320144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.320125    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6sb\" (UniqueName: \"kubernetes.io/projected/28885980-38ee-46ba-9944-e7ce6739496e-kube-api-access-8t6sb\") pod \"iptables-alerter-9wmgw\" (UID: \"28885980-38ee-46ba-9944-e7ce6739496e\") " pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.323437 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.323417    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7qp\" (UniqueName: \"kubernetes.io/projected/8aca6411-d820-4c64-bec2-2ef05551c627-kube-api-access-zr7qp\") pod \"tuned-w692r\" (UID: \"8aca6411-d820-4c64-bec2-2ef05551c627\") " pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.348553 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-system-cni-dir\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.348689 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-kubelet\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348689 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9106ec84-55be-4d4e-be47-97aca4c61dc4-agent-certs\") pod \"konnectivity-agent-29kff\" (UID: \"9106ec84-55be-4d4e-be47-97aca4c61dc4\") " pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.348689 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-system-cni-dir\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.348689 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-node-log\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.348723    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-kubelet\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-log-socket\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348757    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-log-socket\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.348783    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:24.84876413 +0000 UTC m=+3.083718375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-node-log\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-cni-bin\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cnibin\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.348905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-slash\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348918    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-cni-bin\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348944    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-slash\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348961    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cnibin\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348975    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.348975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-run-netns\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-run-netns\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-var-lib-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmwp\" (UniqueName: \"kubernetes.io/projected/0aa73a80-6429-4a23-8445-6b1110dddd5b-kube-api-access-tdmwp\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovn-node-metrics-cert\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovnkube-script-lib\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cni-binary-copy\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.349335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-var-lib-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-systemd\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-cni-netd\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pfs\" (UniqueName: \"kubernetes.io/projected/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-kube-api-access-d7pfs\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-systemd\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovnkube-config\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-env-overrides\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349477    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-cni-netd\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-os-release\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvcw\" (UniqueName: \"kubernetes.io/projected/9f6cb189-4786-4e41-bd91-0888314a0b65-kube-api-access-prvcw\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-systemd-units\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-etc-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-tuning-conf-dir\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349637    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9106ec84-55be-4d4e-be47-97aca4c61dc4-konnectivity-ca\") pod \"konnectivity-agent-29kff\" (UID: \"9106ec84-55be-4d4e-be47-97aca4c61dc4\") " pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.350124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-os-release\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349681    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349699    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-systemd-units\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-etc-openvswitch\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-ovn\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovnkube-script-lib\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349794    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-run-ovn\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349926    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa73a80-6429-4a23-8445-6b1110dddd5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.349991    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-tuning-conf-dir\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.350107    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-env-overrides\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.350166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-cni-binary-copy\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.350411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9106ec84-55be-4d4e-be47-97aca4c61dc4-konnectivity-ca\") pod \"konnectivity-agent-29kff\" (UID: \"9106ec84-55be-4d4e-be47-97aca4c61dc4\") " pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.350760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.350419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovnkube-config\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.351323 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.351244    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9106ec84-55be-4d4e-be47-97aca4c61dc4-agent-certs\") pod \"konnectivity-agent-29kff\" (UID: \"9106ec84-55be-4d4e-be47-97aca4c61dc4\") " pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.352260 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.352241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aa73a80-6429-4a23-8445-6b1110dddd5b-ovn-node-metrics-cert\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.366329 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.366307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pfs\" (UniqueName: \"kubernetes.io/projected/3ebfb66f-330b-495f-bd2b-4c25c6b2f043-kube-api-access-d7pfs\") pod \"multus-additional-cni-plugins-smlvd\" (UID: \"3ebfb66f-330b-495f-bd2b-4c25c6b2f043\") " pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.368885 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.368867    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:24.368885 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.368889    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:24.369042 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.368901    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:24.369042 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.368970    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:24.868950814 +0000 UTC m=+3.103905076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:24.371029 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.371013    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmwp\" (UniqueName: \"kubernetes.io/projected/0aa73a80-6429-4a23-8445-6b1110dddd5b-kube-api-access-tdmwp\") pod \"ovnkube-node-n4bhn\" (UID: \"0aa73a80-6429-4a23-8445-6b1110dddd5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.377119 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.377102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvcw\" (UniqueName: \"kubernetes.io/projected/9f6cb189-4786-4e41-bd91-0888314a0b65-kube-api-access-prvcw\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.441065 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.440974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wqlzd"
Jul 01 18:37:24.449013 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.448953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9wmgw"
Jul 01 18:37:24.460866 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.460843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q"
Jul 01 18:37:24.465098 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.465072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w692r"
Jul 01 18:37:24.470573 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.470549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v6j5f"
Jul 01 18:37:24.476849 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.476835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k5qgm"
Jul 01 18:37:24.484013 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.483997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:24.491904 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.491882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-smlvd"
Jul 01 18:37:24.496993 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.496975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:24.751943 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.751852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:24.752099 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.752006    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:24.752099 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.752086    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:25.752064791 +0000 UTC m=+3.987019051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:24.852559 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.852527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:24.852691 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.852640    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:24.852691 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.852690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:25.852676861 +0000 UTC m=+4.087631106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:24.953790 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:24.953753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:24.953945 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.953925    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:24.954017 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.953950    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:24.954017 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.953962    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:24.954098 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:24.954034    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:25.954013113 +0000 UTC m=+4.188967383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:25.129982 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.129956    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa76cfd_880b_4b2a_a285_e9fc3c4a9a7d.slice/crio-6ffceaf82f6d1e05c071ec11233f42c01173804a30c20337bff51f20722b4d47 WatchSource:0}: Error finding container 6ffceaf82f6d1e05c071ec11233f42c01173804a30c20337bff51f20722b4d47: Status 404 returned error can't find the container with id 6ffceaf82f6d1e05c071ec11233f42c01173804a30c20337bff51f20722b4d47
Jul 01 18:37:25.131198 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.131173    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28885980_38ee_46ba_9944_e7ce6739496e.slice/crio-c39b9c23a1e52bf15aefb3b63892cdbce2d018a82ee3f99d1017fe581901ea8b WatchSource:0}: Error finding container c39b9c23a1e52bf15aefb3b63892cdbce2d018a82ee3f99d1017fe581901ea8b: Status 404 returned error can't find the container with id c39b9c23a1e52bf15aefb3b63892cdbce2d018a82ee3f99d1017fe581901ea8b
Jul 01 18:37:25.132722 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.132672    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00aedd5e_9e98_414c_930d_9d6661e4a0de.slice/crio-79fb581401f76fd90dc1737f1e18adaad97cd4d5d87161b8f403fec38f3a62ae WatchSource:0}: Error finding container 79fb581401f76fd90dc1737f1e18adaad97cd4d5d87161b8f403fec38f3a62ae: Status 404 returned error can't find the container with id 79fb581401f76fd90dc1737f1e18adaad97cd4d5d87161b8f403fec38f3a62ae
Jul 01 18:37:25.134139 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.134113    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aca6411_d820_4c64_bec2_2ef05551c627.slice/crio-96b4dfee84449549bcb459fb6c673bccc7ffbe1dd2692fc45fc4817a85e6b52f WatchSource:0}: Error finding container 96b4dfee84449549bcb459fb6c673bccc7ffbe1dd2692fc45fc4817a85e6b52f: Status 404 returned error can't find the container with id 96b4dfee84449549bcb459fb6c673bccc7ffbe1dd2692fc45fc4817a85e6b52f
Jul 01 18:37:25.140031 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.140010    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9106ec84_55be_4d4e_be47_97aca4c61dc4.slice/crio-429e78c4e45dbfce66f0fa9c92c1dea6b56641ff9057724f9537eb2dee52b243 WatchSource:0}: Error finding container 429e78c4e45dbfce66f0fa9c92c1dea6b56641ff9057724f9537eb2dee52b243: Status 404 returned error can't find the container with id 429e78c4e45dbfce66f0fa9c92c1dea6b56641ff9057724f9537eb2dee52b243
Jul 01 18:37:25.141107 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.141085    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ce6483_0300_475e_855c_e5237856a12f.slice/crio-50effe13c484a02b9315dfb0abdc54b7b1236b5df5e3ee7ac43ee1abe5f8fcae WatchSource:0}: Error finding container 50effe13c484a02b9315dfb0abdc54b7b1236b5df5e3ee7ac43ee1abe5f8fcae: Status 404 returned error can't find the container with id 50effe13c484a02b9315dfb0abdc54b7b1236b5df5e3ee7ac43ee1abe5f8fcae
Jul 01 18:37:25.142156 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.142021    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb73641_462d_4970_98c5_d491835f923f.slice/crio-71bc843f4b119263a2fdf2619e43073870219e1ce181da0284fefe8d0a8e0dc1 WatchSource:0}: Error finding container 71bc843f4b119263a2fdf2619e43073870219e1ce181da0284fefe8d0a8e0dc1: Status 404 returned error can't find the container with id 71bc843f4b119263a2fdf2619e43073870219e1ce181da0284fefe8d0a8e0dc1
Jul 01 18:37:25.142734 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.142717    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa73a80_6429_4a23_8445_6b1110dddd5b.slice/crio-130463e86bf05a0ae4957310c86d9d237b115094e689de8b24b6875fd0019e4d WatchSource:0}: Error finding container 130463e86bf05a0ae4957310c86d9d237b115094e689de8b24b6875fd0019e4d: Status 404 returned error can't find the container with id 130463e86bf05a0ae4957310c86d9d237b115094e689de8b24b6875fd0019e4d
Jul 01 18:37:25.143748 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:25.143520    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebfb66f_330b_495f_bd2b_4c25c6b2f043.slice/crio-52a1044d3d22223112fac519f967936e8a35f3bb8b0fcb75460a21c498f0ecb0 WatchSource:0}: Error finding container 52a1044d3d22223112fac519f967936e8a35f3bb8b0fcb75460a21c498f0ecb0: Status 404 returned error can't find the container with id 52a1044d3d22223112fac519f967936e8a35f3bb8b0fcb75460a21c498f0ecb0
Jul 01 18:37:25.202663 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.202641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:25.202776 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.202757    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:25.210764 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.210743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k5qgm" event={"ID":"9cb73641-462d-4970-98c5-d491835f923f","Type":"ContainerStarted","Data":"71bc843f4b119263a2fdf2619e43073870219e1ce181da0284fefe8d0a8e0dc1"}
Jul 01 18:37:25.211997 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.211971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-29kff" event={"ID":"9106ec84-55be-4d4e-be47-97aca4c61dc4","Type":"ContainerStarted","Data":"429e78c4e45dbfce66f0fa9c92c1dea6b56641ff9057724f9537eb2dee52b243"}
Jul 01 18:37:25.213088 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.213068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w692r" event={"ID":"8aca6411-d820-4c64-bec2-2ef05551c627","Type":"ContainerStarted","Data":"96b4dfee84449549bcb459fb6c673bccc7ffbe1dd2692fc45fc4817a85e6b52f"}
Jul 01 18:37:25.214178 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.214152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q" event={"ID":"00aedd5e-9e98-414c-930d-9d6661e4a0de","Type":"ContainerStarted","Data":"79fb581401f76fd90dc1737f1e18adaad97cd4d5d87161b8f403fec38f3a62ae"}
Jul 01 18:37:25.215402 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.215382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wmgw" event={"ID":"28885980-38ee-46ba-9944-e7ce6739496e","Type":"ContainerStarted","Data":"c39b9c23a1e52bf15aefb3b63892cdbce2d018a82ee3f99d1017fe581901ea8b"}
Jul 01 18:37:25.216465 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.216437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6j5f" event={"ID":"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d","Type":"ContainerStarted","Data":"6ffceaf82f6d1e05c071ec11233f42c01173804a30c20337bff51f20722b4d47"}
Jul 01 18:37:25.217588 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.217569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerStarted","Data":"52a1044d3d22223112fac519f967936e8a35f3bb8b0fcb75460a21c498f0ecb0"}
Jul 01 18:37:25.218660 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.218629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"130463e86bf05a0ae4957310c86d9d237b115094e689de8b24b6875fd0019e4d"}
Jul 01 18:37:25.219770 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.219750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wqlzd" event={"ID":"c9ce6483-0300-475e-855c-e5237856a12f","Type":"ContainerStarted","Data":"50effe13c484a02b9315dfb0abdc54b7b1236b5df5e3ee7ac43ee1abe5f8fcae"}
Jul 01 18:37:25.271524 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.271466    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:32:24 +0000 UTC" deadline="2027-12-05 01:02:47.992700509 +0000 UTC"
Jul 01 18:37:25.271524 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.271519    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12510h25m22.721183674s"
Jul 01 18:37:25.758434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.758344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:25.758600 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.758507    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:25.758600 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.758567    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:27.758548514 +0000 UTC m=+5.993502782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:25.859137 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.858777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:25.859137 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.858950    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:25.859137 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.859028    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:27.859006251 +0000 UTC m=+6.093960498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:25.959541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:25.959500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:25.959720 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.959676    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:25.959720 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.959698    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:25.959720 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.959711    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:25.959904 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:25.959768    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:27.959750011 +0000 UTC m=+6.194704258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:26.208380 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.207517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:26.208380 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:26.207854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:26.208380 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.208217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:26.208380 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:26.208302    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:26.233351 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.232604    2577 generic.go:356] "Generic (PLEG): container finished" podID="469f128e84139c59dc87d077aeeac40f" containerID="e9d414ec9e0645eb39c0f09955abe2470ae33caabb5d26a0c36d918e20504d3e" exitCode=0
Jul 01 18:37:26.233351 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.232681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal" event={"ID":"469f128e84139c59dc87d077aeeac40f","Type":"ContainerDied","Data":"e9d414ec9e0645eb39c0f09955abe2470ae33caabb5d26a0c36d918e20504d3e"}
Jul 01 18:37:26.243908 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.243457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal" event={"ID":"c582d8238a6f3daead6044036f2a5eeb","Type":"ContainerStarted","Data":"5c171f59ef16dd7fea0c2481a83d1858d981ff9cc40962dc85f2947de1543b06"}
Jul 01 18:37:26.272246 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.272192    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 18:32:24 +0000 UTC" deadline="2027-12-24 17:17:17.914150637 +0000 UTC"
Jul 01 18:37:26.272409 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:26.272395    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12982h39m51.641762441s"
Jul 01 18:37:27.202509 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:27.201975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:27.202509 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.202171    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:27.273270 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:27.273237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal" event={"ID":"469f128e84139c59dc87d077aeeac40f","Type":"ContainerStarted","Data":"eafa342b98bb84667a902f8642c7f0e545f611cfef3d61ca626ed304b0ca0ab3"}
Jul 01 18:37:27.296701 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:27.295675    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-57.ec2.internal" podStartSLOduration=4.295654821 podStartE2EDuration="4.295654821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:26.287591426 +0000 UTC m=+4.522545711" watchObservedRunningTime="2026-07-01 18:37:27.295654821 +0000 UTC m=+5.530609088"
Jul 01 18:37:27.776437 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:27.776398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:27.776609 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.776561    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:27.776682 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.776626    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:31.776607244 +0000 UTC m=+10.011561499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:27.878032 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:27.877610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:27.878032 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.877754    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:27.878728 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.878707    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:31.877799602 +0000 UTC m=+10.112753853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:27.978929 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:27.978887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:27.979131 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.979115    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:27.979208 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.979142    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:27.979208 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.979154    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:27.979327 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:27.979227    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:31.979209192 +0000 UTC m=+10.214163462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:28.203757 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:28.203678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:28.203932 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:28.203833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:28.203932 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:28.203836    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:28.204043 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:28.203938    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:29.203064 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:29.202550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:29.203064 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:29.202691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:30.203110 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:30.202990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:30.203110 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:30.202998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:30.203576 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:30.203128    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:30.203576 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:30.203212    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:31.202327 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:31.202294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:31.202504 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:31.202436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:31.812885 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:31.812854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:31.813336 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:31.812994    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:31.813336 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:31.813059    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.813039646 +0000 UTC m=+18.047993906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:31.913291 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:31.913257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:31.913447 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:31.913407    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:31.913525 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:31.913467    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:39.913447612 +0000 UTC m=+18.148401855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:32.014079 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:32.014029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:32.014245 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:32.014202    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:32.014245 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:32.014220    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:32.014245 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:32.014234    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:32.014387 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:32.014292    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:40.014274634 +0000 UTC m=+18.249228893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:32.203841 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:32.203185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:32.203841 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:32.203297    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:32.203841 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:32.203365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:32.203841 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:32.203432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:33.202420 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:33.202296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:33.202885 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:33.202443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:34.206598 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:34.206562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:34.206598 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:34.206599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:34.207098 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:34.206684    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:34.207098 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:34.206745    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:35.202205 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:35.202179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:35.202384 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:35.202280    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:36.202190 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:36.202156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:36.202639 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:36.202156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:36.202639 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:36.202277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:36.202639 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:36.202373    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:37.201943 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:37.201912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:37.202126 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:37.202040    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:38.205877 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:38.205847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:38.206312 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:38.205856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:38.206312 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:38.205961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:38.206312 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:38.206058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:39.202336 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:39.202303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:39.202489 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:39.202448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:39.873699 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:39.873660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:39.874211 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:39.873846    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:39.874211 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:39.873925    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:55.873903249 +0000 UTC m=+34.108857510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:39.974357 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:39.974323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:39.974541 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:39.974456    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:39.974541 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:39.974522    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:55.974503685 +0000 UTC m=+34.209457943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:40.075547 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:40.075513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:40.075732 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:40.075710    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:40.075786 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:40.075738    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:40.075786 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:40.075751    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:40.075908 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:40.075829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:56.075795801 +0000 UTC m=+34.310750049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:40.202735 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:40.202660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:40.202924 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:40.202660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:40.202924 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:40.202787    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:40.202924 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:40.202876    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:41.202422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:41.202385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:41.202858 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:41.202534    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:42.204914 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:42.204887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:42.205360 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:42.204989    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:42.205360 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:42.205051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:42.205360 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:42.205148    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:43.202254 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:43.202224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:43.202441 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:43.202332    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:44.202979 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.202957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:44.203319 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.202986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:44.203319 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:44.203063    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:44.203319 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:44.203180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:44.310825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.310663    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb66f-330b-495f-bd2b-4c25c6b2f043" containerID="a96c6a8d5caabf9cc5e5fb97d84963741c107fd3b477dc1d599ed4919188d2cd" exitCode=0
Jul 01 18:37:44.311001 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.310747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerDied","Data":"a96c6a8d5caabf9cc5e5fb97d84963741c107fd3b477dc1d599ed4919188d2cd"}
Jul 01 18:37:44.313251 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.313231    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:37:44.313628 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.313605    2577 generic.go:356] "Generic (PLEG): container finished" podID="0aa73a80-6429-4a23-8445-6b1110dddd5b" containerID="7e664c3486797101a475e2a33a28fcf4ef1d959ca629dd44d54d3ab13177d09e" exitCode=1
Jul 01 18:37:44.313719 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.313677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"c41259f73d0fc9f3b917c9c1cf5fcf7bbeacad551f0f52c7874fbed70f68709e"}
Jul 01 18:37:44.313780 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.313771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerDied","Data":"7e664c3486797101a475e2a33a28fcf4ef1d959ca629dd44d54d3ab13177d09e"}
Jul 01 18:37:44.313858 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.313838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"89ac273cf7d262c2576fb3563ff527354c73fb0e2957529b9c1831b21c22a492"}
Jul 01 18:37:44.316080 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.316034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wqlzd" event={"ID":"c9ce6483-0300-475e-855c-e5237856a12f","Type":"ContainerStarted","Data":"787a40b8f4fd9c084a54e61786bd7dc8236853d23cf928a7f92972c89f3fbd21"}
Jul 01 18:37:44.318840 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.318796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k5qgm" event={"ID":"9cb73641-462d-4970-98c5-d491835f923f","Type":"ContainerStarted","Data":"611cb04166c74e79bf37d2f3db26fa7b8c095d575936eb78bf8797c473190cfd"}
Jul 01 18:37:44.320412 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.320387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-29kff" event={"ID":"9106ec84-55be-4d4e-be47-97aca4c61dc4","Type":"ContainerStarted","Data":"a3218fb36bd03e9480d4a2b828ca8fcf5071bd48e458b92040e82b45f86691ae"}
Jul 01 18:37:44.322062 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.322039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w692r" event={"ID":"8aca6411-d820-4c64-bec2-2ef05551c627","Type":"ContainerStarted","Data":"62aeb53f9653415a1d51527e5eed5c121ae9d93ba10652ccc2a1ff5bd8d2f30a"}
Jul 01 18:37:44.323851 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.323832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q" event={"ID":"00aedd5e-9e98-414c-930d-9d6661e4a0de","Type":"ContainerStarted","Data":"6c6c48b2437386901149145729426e70e42b38ced554119999a0a4abb11fffec"}
Jul 01 18:37:44.325857 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.325787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v6j5f" event={"ID":"1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d","Type":"ContainerStarted","Data":"896ec4277d5a89a541b1df9518e4689cbd5428cc6eb040f0c15b8641bb6f7256"}
Jul 01 18:37:44.362785 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.362726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-57.ec2.internal" podStartSLOduration=21.362706199 podStartE2EDuration="21.362706199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:27.296645454 +0000 UTC m=+5.531599741" watchObservedRunningTime="2026-07-01 18:37:44.362706199 +0000 UTC m=+22.597660468"
Jul 01 18:37:44.445547 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.445508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v6j5f" podStartSLOduration=3.898730778 podStartE2EDuration="22.445489589s" totalImagesPullingTime="18.546758811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.44528683 +0000 UTC m=+22.680241117" watchObservedRunningTime="2026-07-01 18:37:44.445489589 +0000 UTC m=+22.680443857"
Jul 01 18:37:44.445793 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.445761    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wqlzd" podStartSLOduration=3.896338965 podStartE2EDuration="22.445751949s" totalImagesPullingTime="18.549412984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.400597231 +0000 UTC m=+22.635551497" watchObservedRunningTime="2026-07-01 18:37:44.445751949 +0000 UTC m=+22.680706220"
Jul 01 18:37:44.473199 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.473166    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w692r" podStartSLOduration=3.930186547 podStartE2EDuration="22.473152295s" totalImagesPullingTime="18.542965748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.471976065 +0000 UTC m=+22.706930331" watchObservedRunningTime="2026-07-01 18:37:44.473152295 +0000 UTC m=+22.708106561"
Jul 01 18:37:44.561761 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.561661    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k5qgm" podStartSLOduration=4.0265909539999996 podStartE2EDuration="22.561641897s" totalImagesPullingTime="18.535050943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.502317039 +0000 UTC m=+22.737271305" watchObservedRunningTime="2026-07-01 18:37:44.561641897 +0000 UTC m=+22.796596166"
Jul 01 18:37:44.650720 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:44.650685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:45.049405 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.049380    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 18:37:45.201988 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.201916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:45.202140 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:45.202022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:45.204020 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.203943    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T18:37:45.049399451Z","UUID":"721328d0-f3b7-4df4-91b4-75820c3f875b","Handler":null,"Name":"","Endpoint":""}
Jul 01 18:37:45.205389 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.205374    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 18:37:45.205478 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.205404    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 18:37:45.330342 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.330268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q" event={"ID":"00aedd5e-9e98-414c-930d-9d6661e4a0de","Type":"ContainerStarted","Data":"3e528eded54b3e19929f4151dedbfb1bafd528f0cfea118e304a8e22b8f888e1"}
Jul 01 18:37:45.332077 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.331876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9wmgw" event={"ID":"28885980-38ee-46ba-9944-e7ce6739496e","Type":"ContainerStarted","Data":"75afa2550aba69e5f91690d73ea8abdb0ed6c9960a731401e034236612663596"}
Jul 01 18:37:45.334823 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.334790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:37:45.335312 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.335287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"d97f6ab9db0d89cf9fcf9814bad3e966dad5151874f523081e3b9446ec065faf"}
Jul 01 18:37:45.335436 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.335365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"19fff0d223ba368735bf6b285f25580326ce7a544b52b6287b9a2d36240fe788"}
Jul 01 18:37:45.335436 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.335413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"ec429f449fd57be982419e6e92d4c4b54ab1a3d97823294c9b4c4d7ff3bfa0ba"}
Jul 01 18:37:45.379484 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.379427    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-29kff" podStartSLOduration=9.521840239 podStartE2EDuration="23.379407308s" totalImagesPullingTime="13.857567069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:44.562270837 +0000 UTC m=+22.797225127" watchObservedRunningTime="2026-07-01 18:37:45.379407308 +0000 UTC m=+23.614361615"
Jul 01 18:37:45.845720 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:45.845688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:46.198563 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:46.198535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:46.202026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:46.202003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:46.202142 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:46.202120    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:46.202246 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:46.202221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:46.202350 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:46.202329    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:46.339384 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:46.339329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q" event={"ID":"00aedd5e-9e98-414c-930d-9d6661e4a0de","Type":"ContainerStarted","Data":"625137a5da51ae5467edf0bb2dcf8c2a41a189ad3d9a23fe75d028c3b66dac93"}
Jul 01 18:37:46.362018 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:46.361983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:46.374188 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:46.374137    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9wmgw" podStartSLOduration=5.851818086 podStartE2EDuration="24.374115514s" totalImagesPullingTime="18.522297428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:45.405690807 +0000 UTC m=+23.640645074" watchObservedRunningTime="2026-07-01 18:37:46.374115514 +0000 UTC m=+24.609069785"
Jul 01 18:37:47.202401 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:47.202375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:47.202560 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:47.202476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:47.345021 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:47.344997    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:37:47.345421 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:47.345392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"da063461c82dec90395252ca0e015f081331eaaaa371c9e7934f45a7e7635027"}
Jul 01 18:37:48.203042 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:48.203004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:48.203042 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:48.203029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:48.203269 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:48.203118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:48.203327 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:48.203290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:48.622219 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:48.622136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:48.622850 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:48.622799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:48.652753 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:48.652705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86f7q" podStartSLOduration=5.985182903 podStartE2EDuration="26.652688532s" totalImagesPullingTime="20.667505629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:46.373857393 +0000 UTC m=+24.608811681" watchObservedRunningTime="2026-07-01 18:37:48.652688532 +0000 UTC m=+26.887642797"
Jul 01 18:37:49.202204 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.201986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:49.202335 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:49.202253    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:49.351373 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.351346    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb66f-330b-495f-bd2b-4c25c6b2f043" containerID="e75b8b567d046c36877702484362a234c6574ce9f707a9295df9044f7457234f" exitCode=0
Jul 01 18:37:49.351536 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.351430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerDied","Data":"e75b8b567d046c36877702484362a234c6574ce9f707a9295df9044f7457234f"}
Jul 01 18:37:49.354645 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.354631    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:37:49.355026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.355007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"12e4ea8fd07d5a6f436a1f6e839332513205051c5578aceacb61ce72cec279fb"}
Jul 01 18:37:49.355232 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.355212    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:49.355373 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.355360    2577 scope.go:117] "RemoveContainer" containerID="7e664c3486797101a475e2a33a28fcf4ef1d959ca629dd44d54d3ab13177d09e"
Jul 01 18:37:49.355613 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.355598    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-29kff"
Jul 01 18:37:49.593499 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:49.593269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:50.202747 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.202713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:50.203444 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:50.202864    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:50.203444 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.202922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:50.203444 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:50.203036    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:50.360059 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.359994    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:37:50.360288 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.360269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" event={"ID":"0aa73a80-6429-4a23-8445-6b1110dddd5b","Type":"ContainerStarted","Data":"ce56be9df31917787c75a5af70bc11b0553a54aaddc7d0c5bf1341660b75c9d2"}
Jul 01 18:37:50.360535 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.360512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:50.360615 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.360568    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:50.360665 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.360616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:50.375415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.375382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:50.375537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.375489    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:37:50.502566 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:50.502514    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn" podStartSLOduration=9.71031388 podStartE2EDuration="28.502499385s" totalImagesPullingTime="18.792185505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:37:50.452323263 +0000 UTC m=+28.687277550" watchObservedRunningTime="2026-07-01 18:37:50.502499385 +0000 UTC m=+28.737453650"
Jul 01 18:37:51.202508 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.202472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:51.202702 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:51.202585    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:51.338442 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.338408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c69kt"]
Jul 01 18:37:51.338848 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.338515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:51.338848 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:51.338589    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:51.342649 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.342623    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkqt7"]
Jul 01 18:37:51.348669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.348651    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k499p"]
Jul 01 18:37:51.348778 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.348742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:51.348864 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:51.348846    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:51.364795 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.364707    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb66f-330b-495f-bd2b-4c25c6b2f043" containerID="ae4da6c6b52b2285bc0df36d2419cf8c10fdf61c7d4095d58e4a88317e0a475e" exitCode=0
Jul 01 18:37:51.364795 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.364790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:51.365004 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:51.364830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerDied","Data":"ae4da6c6b52b2285bc0df36d2419cf8c10fdf61c7d4095d58e4a88317e0a475e"}
Jul 01 18:37:51.365306 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:51.365276    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:53.202945 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:53.202875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:53.202945 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:53.202890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:53.202945 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:53.202890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:53.203316 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:53.202979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:53.203316 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:53.203074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:53.203316 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:53.203147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:53.371040 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:53.371008    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb66f-330b-495f-bd2b-4c25c6b2f043" containerID="e1961ac0d6623612015960772f8ba2c6360b6e95d05cca3985703d1961f12223" exitCode=0
Jul 01 18:37:53.371172 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:53.371085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerDied","Data":"e1961ac0d6623612015960772f8ba2c6360b6e95d05cca3985703d1961f12223"}
Jul 01 18:37:55.202780 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:55.202749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:55.203239 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:55.202749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:55.203239 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.202872    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:55.203239 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:55.202754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:55.203239 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.202928    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:55.203239 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.202996    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:55.890624 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:55.890579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:55.890836 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.890824    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:55.890902 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.890892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret podName:cba52558-8c44-4d21-94cf-ed28570c6337 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.890865482 +0000 UTC m=+66.125819730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret") pod "global-pull-secret-syncer-k499p" (UID: "cba52558-8c44-4d21-94cf-ed28570c6337") : object "kube-system"/"original-pull-secret" not registered
Jul 01 18:37:55.991190 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:55.991159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:55.991341 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.991254    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:55.991341 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:55.991300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:27.991287768 +0000 UTC m=+66.226242012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 18:37:56.091469 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:56.091439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:56.091646 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:56.091595    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 18:37:56.091646 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:56.091616    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 18:37:56.091646 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:56.091627    2577 projected.go:196] Error preparing data for projected volume kube-api-access-svjmq for pod openshift-network-diagnostics/network-check-target-c69kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:56.091825 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:56.091685    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq podName:11574496-4380-4df8-a38b-5e732c84d264 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.091666282 +0000 UTC m=+66.326620526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-svjmq" (UniqueName: "kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq") pod "network-check-target-c69kt" (UID: "11574496-4380-4df8-a38b-5e732c84d264") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 18:37:57.202118 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.202084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:57.202719 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.202201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:57.202719 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:57.202215    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k499p" podUID="cba52558-8c44-4d21-94cf-ed28570c6337"
Jul 01 18:37:57.202719 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.202241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:57.202719 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:57.202334    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c69kt" podUID="11574496-4380-4df8-a38b-5e732c84d264"
Jul 01 18:37:57.202719 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:57.202443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xkqt7" podUID="9f6cb189-4786-4e41-bd91-0888314a0b65"
Jul 01 18:37:57.615153 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.615077    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-57.ec2.internal" event="NodeReady"
Jul 01 18:37:57.615300 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.615209    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 18:37:57.729464 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.729433    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9s8cn"]
Jul 01 18:37:57.732315 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.732291    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-skvhh"]
Jul 01 18:37:57.732445 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.732430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:57.734089 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.734067    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:57.737071 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.737052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4nfs\""
Jul 01 18:37:57.737166 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.737113    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 18:37:57.737485 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.737468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 18:37:57.737721 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.737707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 18:37:57.737827 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.737738    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 18:37:57.738043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.738025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w67vh\""
Jul 01 18:37:57.738102 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.738092    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 18:37:57.772623 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.772600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9s8cn"]
Jul 01 18:37:57.820539 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.820498    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-skvhh"]
Jul 01 18:37:57.906511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.906420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcnt\" (UniqueName: \"kubernetes.io/projected/6b9359a6-a671-416d-89cd-fc18afd1fdc8-kube-api-access-dbcnt\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:57.906511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.906459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:57.906511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.906475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9359a6-a671-416d-89cd-fc18afd1fdc8-tmp-dir\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:57.906511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.906499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d13004e-5a72-4440-8c2d-df5ea9e56024-cert\") pod \"ingress-canary-skvhh\" (UID: \"6d13004e-5a72-4440-8c2d-df5ea9e56024\") " pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:57.906765 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.906532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9359a6-a671-416d-89cd-fc18afd1fdc8-config-volume\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:57.906765 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:57.906575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxmz\" (UniqueName: \"kubernetes.io/projected/6d13004e-5a72-4440-8c2d-df5ea9e56024-kube-api-access-svxmz\") pod \"ingress-canary-skvhh\" (UID: \"6d13004e-5a72-4440-8c2d-df5ea9e56024\") " pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:58.007358 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.007319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9359a6-a671-416d-89cd-fc18afd1fdc8-config-volume\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.007358 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.007365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxmz\" (UniqueName: \"kubernetes.io/projected/6d13004e-5a72-4440-8c2d-df5ea9e56024-kube-api-access-svxmz\") pod \"ingress-canary-skvhh\" (UID: \"6d13004e-5a72-4440-8c2d-df5ea9e56024\") " pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:58.007611 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.007449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcnt\" (UniqueName: \"kubernetes.io/projected/6b9359a6-a671-416d-89cd-fc18afd1fdc8-kube-api-access-dbcnt\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.007611 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.007481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.007611 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.007505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9359a6-a671-416d-89cd-fc18afd1fdc8-tmp-dir\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.007611 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.007530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d13004e-5a72-4440-8c2d-df5ea9e56024-cert\") pod \"ingress-canary-skvhh\" (UID: \"6d13004e-5a72-4440-8c2d-df5ea9e56024\") " pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:58.007831 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:58.007682    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:37:58.007831 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:58.007739    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:58.507721467 +0000 UTC m=+36.742675714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:37:58.008023 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.008004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6b9359a6-a671-416d-89cd-fc18afd1fdc8-config-volume\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.008106 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.008084    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9359a6-a671-416d-89cd-fc18afd1fdc8-tmp-dir\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.012071 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.012047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d13004e-5a72-4440-8c2d-df5ea9e56024-cert\") pod \"ingress-canary-skvhh\" (UID: \"6d13004e-5a72-4440-8c2d-df5ea9e56024\") " pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:58.030855 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.030799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxmz\" (UniqueName: \"kubernetes.io/projected/6d13004e-5a72-4440-8c2d-df5ea9e56024-kube-api-access-svxmz\") pod \"ingress-canary-skvhh\" (UID: \"6d13004e-5a72-4440-8c2d-df5ea9e56024\") " pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:58.035648 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.035584    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcnt\" (UniqueName: \"kubernetes.io/projected/6b9359a6-a671-416d-89cd-fc18afd1fdc8-kube-api-access-dbcnt\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.050550 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.050517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-skvhh"
Jul 01 18:37:58.172352 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.172277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:58.311682 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.311647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:58.472617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.472418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-skvhh"]
Jul 01 18:37:58.474668 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:37:58.474645    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d13004e_5a72_4440_8c2d_df5ea9e56024.slice/crio-2aaaf525bc32281099a5591399ffa8109c7584011ae13a6107f05f969b1d2f19 WatchSource:0}: Error finding container 2aaaf525bc32281099a5591399ffa8109c7584011ae13a6107f05f969b1d2f19: Status 404 returned error can't find the container with id 2aaaf525bc32281099a5591399ffa8109c7584011ae13a6107f05f969b1d2f19
Jul 01 18:37:58.512193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.512167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:58.512371 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:58.512339    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:37:58.512430 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:58.512411    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:37:59.512391025 +0000 UTC m=+37.747345283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:37:58.771304 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.771199    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:58.914383 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:58.914349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:59.054919 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.054843    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:37:59.202533 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.202500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:37:59.202711 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.202500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:37:59.202711 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.202500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:37:59.210558 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.210061    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:37:59.211073 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.211050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:37:59.211168 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.211073    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:37:59.211239 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.211190    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dgh4l\""
Jul 01 18:37:59.211239 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.211195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plpqm\""
Jul 01 18:37:59.211239 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.211204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:37:59.389671 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.389602    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb66f-330b-495f-bd2b-4c25c6b2f043" containerID="2307b989f40ee7862e046e2c3c506a8cc92d2e9ba140c87f6202bb26fbb16942" exitCode=0
Jul 01 18:37:59.390100 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.389671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerDied","Data":"2307b989f40ee7862e046e2c3c506a8cc92d2e9ba140c87f6202bb26fbb16942"}
Jul 01 18:37:59.391154 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.391115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-skvhh" event={"ID":"6d13004e-5a72-4440-8c2d-df5ea9e56024","Type":"ContainerStarted","Data":"2aaaf525bc32281099a5591399ffa8109c7584011ae13a6107f05f969b1d2f19"}
Jul 01 18:37:59.521906 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:37:59.521503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:37:59.521906 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:59.521774    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:37:59.521906 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:37:59.521861    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:01.521838728 +0000 UTC m=+39.756792992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:00.397167 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:00.397137    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebfb66f-330b-495f-bd2b-4c25c6b2f043" containerID="12b4b7c002ec3d327224bf16adf632b616700447183a02205acab3590c0daad8" exitCode=0
Jul 01 18:38:00.397656 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:00.397219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerDied","Data":"12b4b7c002ec3d327224bf16adf632b616700447183a02205acab3590c0daad8"}
Jul 01 18:38:01.101131 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.101099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:01.259598 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.259565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:01.400636 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.400600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-skvhh" event={"ID":"6d13004e-5a72-4440-8c2d-df5ea9e56024","Type":"ContainerStarted","Data":"99578a299a59038502f5920f2f1299171fa2a546192633e4c13e7eecad6a4dd5"}
Jul 01 18:38:01.403781 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.403760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-smlvd" event={"ID":"3ebfb66f-330b-495f-bd2b-4c25c6b2f043","Type":"ContainerStarted","Data":"d9243d1da20079d27318470706638f66b91392e19e58cf40a7f2cb9e7adce288"}
Jul 01 18:38:01.432829 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.430457    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-skvhh" podStartSLOduration=1.844028315 podStartE2EDuration="4.430437783s" totalImagesPullingTime="2.586409468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:01.429766909 +0000 UTC m=+39.664721186" watchObservedRunningTime="2026-07-01 18:38:01.430437783 +0000 UTC m=+39.665392051"
Jul 01 18:38:01.534253 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.534198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:38:01.534425 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:01.534396    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:01.534676 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:01.534467    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:05.534446472 +0000 UTC m=+43.769400737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:01.701158 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.701086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:01.823569 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:01.823545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:05.561235 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:05.561194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:38:05.561628 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:05.561342    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:05.561628 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:05.561406    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:13.561389666 +0000 UTC m=+51.796343910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:13.618127 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:13.618086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:38:13.618617 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:13.618216    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:13.618617 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:13.618273    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:29.618260547 +0000 UTC m=+67.853214791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:18.315308 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.315251    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-smlvd" podStartSLOduration=26.087342651 podStartE2EDuration="56.315233832s" totalImagesPullingTime="30.227891181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:01.467003446 +0000 UTC m=+39.701957733" watchObservedRunningTime="2026-07-01 18:38:18.315233832 +0000 UTC m=+56.550188098"
Jul 01 18:38:18.315883 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.315765    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"]
Jul 01 18:38:18.317771 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.317750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.325235 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.325212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 01 18:38:18.325235 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.325229    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m76ht\""
Jul 01 18:38:18.325392 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.325265    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 01 18:38:18.325392 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.325268    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 18:38:18.325392 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.325304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 18:38:18.336189 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.336169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"]
Jul 01 18:38:18.450418 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.450389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/728e3048-f973-425d-9c5f-27a99f6295b5-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.450418 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.450420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwhn\" (UniqueName: \"kubernetes.io/projected/728e3048-f973-425d-9c5f-27a99f6295b5-kube-api-access-ltwhn\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.450615 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.450477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.551377 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.551338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.551531 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.551432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/728e3048-f973-425d-9c5f-27a99f6295b5-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.551531 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.551450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwhn\" (UniqueName: \"kubernetes.io/projected/728e3048-f973-425d-9c5f-27a99f6295b5-kube-api-access-ltwhn\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.551531 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:18.551492    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:18.551627 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:18.551572    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls podName:728e3048-f973-425d-9c5f-27a99f6295b5 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:19.051554352 +0000 UTC m=+57.286508601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-5dvqm" (UID: "728e3048-f973-425d-9c5f-27a99f6295b5") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:18.552151 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.552133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/728e3048-f973-425d-9c5f-27a99f6295b5-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.575349 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.575286    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwhn\" (UniqueName: \"kubernetes.io/projected/728e3048-f973-425d-9c5f-27a99f6295b5-kube-api-access-ltwhn\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:18.792318 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.792287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:18.911978 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:18.911909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:19.055085 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:19.055056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:19.055207 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:19.055194    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:19.055270 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:19.055261    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls podName:728e3048-f973-425d-9c5f-27a99f6295b5 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:20.055246499 +0000 UTC m=+58.290200742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-5dvqm" (UID: "728e3048-f973-425d-9c5f-27a99f6295b5") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:20.062313 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:20.062264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:20.062701 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:20.062412    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:20.062701 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:20.062472    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls podName:728e3048-f973-425d-9c5f-27a99f6295b5 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:22.062455992 +0000 UTC m=+60.297410249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-5dvqm" (UID: "728e3048-f973-425d-9c5f-27a99f6295b5") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:22.078310 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.078268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:22.078702 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:22.078428    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:22.078702 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:22.078518    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls podName:728e3048-f973-425d-9c5f-27a99f6295b5 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:26.078499126 +0000 UTC m=+64.313453371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-5dvqm" (UID: "728e3048-f973-425d-9c5f-27a99f6295b5") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:22.386828 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.386749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4bhn"
Jul 01 18:38:22.664330 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.664256    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gpcfz"]
Jul 01 18:38:22.696422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.696395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.729228 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.729200    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 18:38:22.733323 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.733299    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 18:38:22.738772 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.738758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 18:38:22.759995 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.759975    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 18:38:22.760350 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.760334    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pp2wl\""
Jul 01 18:38:22.783056 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.783030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.783143 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.783085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.783143 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.783112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfjd\" (UniqueName: \"kubernetes.io/projected/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-api-access-msfjd\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.783143 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.783129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/375d8359-feea-4c7a-b45e-169f4badc3e4-data-volume\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.783236 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.783166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/375d8359-feea-4c7a-b45e-169f4badc3e4-crio-socket\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884277 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfjd\" (UniqueName: \"kubernetes.io/projected/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-api-access-msfjd\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884277 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/375d8359-feea-4c7a-b45e-169f4badc3e4-data-volume\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884463 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/375d8359-feea-4c7a-b45e-169f4badc3e4-crio-socket\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884463 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884463 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884463 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884447    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/375d8359-feea-4c7a-b45e-169f4badc3e4-crio-socket\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.884617 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:22.884474    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:22.884617 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:22.884534    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:23.384520573 +0000 UTC m=+61.619474816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:22.884617 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:22.884475    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:22.884617 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:22.884612    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:23.384599323 +0000 UTC m=+61.619553572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:22.884758 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.884664    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/375d8359-feea-4c7a-b45e-169f4badc3e4-data-volume\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:22.894637 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.894608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gpcfz"]
Jul 01 18:38:22.960339 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:22.960258    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfjd\" (UniqueName: \"kubernetes.io/projected/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-api-access-msfjd\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:23.137997 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:23.137961    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:23.389327 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:23.389286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:23.389529 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:23.389371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:23.389529 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:23.389416    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:23.389529 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:23.389489    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:23.389661 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:23.389547    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:24.389523771 +0000 UTC m=+62.624478015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:23.389661 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:23.389585    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:24.389573058 +0000 UTC m=+62.624527305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:23.409332 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:23.409305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:23.596818 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:23.596775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:24.395955 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:24.395920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:24.396458 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:24.395990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:24.396458 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:24.396084    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:24.396458 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:24.396144    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:26.396129545 +0000 UTC m=+64.631083789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:24.396458 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:24.396089    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:24.396458 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:24.396235    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:26.396217875 +0000 UTC m=+64.631172119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:24.810688 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:24.810666    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6j5f_1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d/dns-node-resolver/0.log"
Jul 01 18:38:25.418376 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:25.418350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k5qgm_9cb73641-462d-4970-98c5-d491835f923f/node-ca/0.log"
Jul 01 18:38:26.107668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:26.107611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:26.107874 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:26.107762    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:26.107874 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:26.107858    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls podName:728e3048-f973-425d-9c5f-27a99f6295b5 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:34.107842126 +0000 UTC m=+72.342796369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-5dvqm" (UID: "728e3048-f973-425d-9c5f-27a99f6295b5") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:26.409250 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:26.409165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:26.409414 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:26.409265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:26.409414 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:26.409283    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:26.409414 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:26.409356    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:30.409340927 +0000 UTC m=+68.644295174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:26.409414 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:26.409391    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:26.409576 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:26.409449    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:30.409433058 +0000 UTC m=+68.644387319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:26.416603 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:26.416583    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-skvhh_6d13004e-5a72-4440-8c2d-df5ea9e56024/serve-healthcheck-canary/0.log"
Jul 01 18:38:27.228152 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.228118    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"]
Jul 01 18:38:27.230673 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.230658    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"
Jul 01 18:38:27.258179 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.258162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:38:27.258286 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.258269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dlvlz\""
Jul 01 18:38:27.286703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.286679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 01 18:38:27.291427 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.291409    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"]
Jul 01 18:38:27.315991 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.315968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqqh\" (UniqueName: \"kubernetes.io/projected/a836d8e1-cc91-4cf3-ada8-b1b5955b63dc-kube-api-access-6gqqh\") pod \"volume-data-source-validator-69d9f679db-cmldk\" (UID: \"a836d8e1-cc91-4cf3-ada8-b1b5955b63dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"
Jul 01 18:38:27.366865 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.366835    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"]
Jul 01 18:38:27.368869 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.368855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:27.371569 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.371546    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 01 18:38:27.372006 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.371989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:38:27.372096 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.371995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vlx2t\""
Jul 01 18:38:27.372213 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.372198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 01 18:38:27.416879 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.416854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:27.417029 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.416949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdwb\" (UniqueName: \"kubernetes.io/projected/b5ee0e9e-2392-45b1-af1f-772e235cafe7-kube-api-access-lbdwb\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:27.417099 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.417033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqqh\" (UniqueName: \"kubernetes.io/projected/a836d8e1-cc91-4cf3-ada8-b1b5955b63dc-kube-api-access-6gqqh\") pod \"volume-data-source-validator-69d9f679db-cmldk\" (UID: \"a836d8e1-cc91-4cf3-ada8-b1b5955b63dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"
Jul 01 18:38:27.467112 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.467083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqqh\" (UniqueName: \"kubernetes.io/projected/a836d8e1-cc91-4cf3-ada8-b1b5955b63dc-kube-api-access-6gqqh\") pod \"volume-data-source-validator-69d9f679db-cmldk\" (UID: \"a836d8e1-cc91-4cf3-ada8-b1b5955b63dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"
Jul 01 18:38:27.482814 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.482736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"]
Jul 01 18:38:27.518292 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.518267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:27.518424 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.518322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdwb\" (UniqueName: \"kubernetes.io/projected/b5ee0e9e-2392-45b1-af1f-772e235cafe7-kube-api-access-lbdwb\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:27.518490 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:27.518425    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:27.518546 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:27.518496    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls podName:b5ee0e9e-2392-45b1-af1f-772e235cafe7 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:28.018475213 +0000 UTC m=+66.253429471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-bjs8g" (UID: "b5ee0e9e-2392-45b1-af1f-772e235cafe7") : secret "samples-operator-tls" not found
Jul 01 18:38:27.537795 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.537770    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdwb\" (UniqueName: \"kubernetes.io/projected/b5ee0e9e-2392-45b1-af1f-772e235cafe7-kube-api-access-lbdwb\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:27.539586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.539566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"
Jul 01 18:38:27.678044 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.678015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk"]
Jul 01 18:38:27.682046 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:27.682019    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda836d8e1_cc91_4cf3_ada8_b1b5955b63dc.slice/crio-c9a2a96d6dd580f0016802834a83bf62938335f79cee32de25aa7986fa786a79 WatchSource:0}: Error finding container c9a2a96d6dd580f0016802834a83bf62938335f79cee32de25aa7986fa786a79: Status 404 returned error can't find the container with id c9a2a96d6dd580f0016802834a83bf62938335f79cee32de25aa7986fa786a79
Jul 01 18:38:27.921103 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.921026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:38:27.924129 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.924111    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 18:38:27.933276 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:27.933249    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cba52558-8c44-4d21-94cf-ed28570c6337-original-pull-secret\") pod \"global-pull-secret-syncer-k499p\" (UID: \"cba52558-8c44-4d21-94cf-ed28570c6337\") " pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:38:28.014516 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.014483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k499p"
Jul 01 18:38:28.021418 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.021397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:28.021521 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.021467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:38:28.021577 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:28.021544    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:28.021625 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:28.021615    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls podName:b5ee0e9e-2392-45b1-af1f-772e235cafe7 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:29.021592745 +0000 UTC m=+67.256546990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-bjs8g" (UID: "b5ee0e9e-2392-45b1-af1f-772e235cafe7") : secret "samples-operator-tls" not found
Jul 01 18:38:28.025773 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.025753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 18:38:28.031917 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:28.031902    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 18:38:28.032003 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:28.031960    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs podName:9f6cb189-4786-4e41-bd91-0888314a0b65 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:32.031940684 +0000 UTC m=+130.266894944 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs") pod "network-metrics-daemon-xkqt7" (UID: "9f6cb189-4786-4e41-bd91-0888314a0b65") : secret "metrics-daemon-secret" not found
Jul 01 18:38:28.122684 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.122622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:38:28.125456 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.125438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 18:38:28.139652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.139631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 18:38:28.146924 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.146902    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjmq\" (UniqueName: \"kubernetes.io/projected/11574496-4380-4df8-a38b-5e732c84d264-kube-api-access-svjmq\") pod \"network-check-target-c69kt\" (UID: \"11574496-4380-4df8-a38b-5e732c84d264\") " pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:38:28.157221 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.157192    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k499p"]
Jul 01 18:38:28.160554 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:28.160532    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba52558_8c44_4d21_94cf_ed28570c6337.slice/crio-db43c5bcb29bd5a5eb4cecc6c9bee855eda45c23b2f98744a9819fec63481405 WatchSource:0}: Error finding container db43c5bcb29bd5a5eb4cecc6c9bee855eda45c23b2f98744a9819fec63481405: Status 404 returned error can't find the container with id db43c5bcb29bd5a5eb4cecc6c9bee855eda45c23b2f98744a9819fec63481405
Jul 01 18:38:28.296014 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.295978    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-jnztt"]
Jul 01 18:38:28.298279 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.298261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.305166 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.304965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 01 18:38:28.306583 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.306567    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:38:28.306677 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.306594    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 01 18:38:28.311557 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.311155    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 01 18:38:28.315159 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.315143    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jsqxc\""
Jul 01 18:38:28.323090 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.323072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 01 18:38:28.324215 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.324193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgrd\" (UniqueName: \"kubernetes.io/projected/101b5c81-fe81-4627-9485-a46d1c8844b1-kube-api-access-plgrd\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.324305 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.324285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/101b5c81-fe81-4627-9485-a46d1c8844b1-config\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.324447 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.324428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/101b5c81-fe81-4627-9485-a46d1c8844b1-serving-cert\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.324522 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.324476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/101b5c81-fe81-4627-9485-a46d1c8844b1-trusted-ca\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.324582 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.324566    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dgh4l\""
Jul 01 18:38:28.331874 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.331855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:38:28.340471 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.340452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-jnztt"]
Jul 01 18:38:28.398639 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.398610    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"]
Jul 01 18:38:28.403301 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.403142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.407398 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.407202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 01 18:38:28.407840 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.407572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 01 18:38:28.407840 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.407659    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gfgbp\""
Jul 01 18:38:28.414052 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.414033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 01 18:38:28.414304 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.414290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 01 18:38:28.425437 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/101b5c81-fe81-4627-9485-a46d1c8844b1-config\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.425534 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrr9\" (UniqueName: \"kubernetes.io/projected/4ca4e42f-afd3-466e-9b65-8958c18336a2-kube-api-access-dzrr9\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.425591 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/101b5c81-fe81-4627-9485-a46d1c8844b1-serving-cert\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.425647 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/101b5c81-fe81-4627-9485-a46d1c8844b1-trusted-ca\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.425701 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca4e42f-afd3-466e-9b65-8958c18336a2-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.425754 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ca4e42f-afd3-466e-9b65-8958c18336a2-config\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.425754 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.425747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgrd\" (UniqueName: \"kubernetes.io/projected/101b5c81-fe81-4627-9485-a46d1c8844b1-kube-api-access-plgrd\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.426142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.426125    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/101b5c81-fe81-4627-9485-a46d1c8844b1-config\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.426701 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.426679    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/101b5c81-fe81-4627-9485-a46d1c8844b1-trusted-ca\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.428055 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.428040    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/101b5c81-fe81-4627-9485-a46d1c8844b1-serving-cert\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.464950 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.464914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k499p" event={"ID":"cba52558-8c44-4d21-94cf-ed28570c6337","Type":"ContainerStarted","Data":"db43c5bcb29bd5a5eb4cecc6c9bee855eda45c23b2f98744a9819fec63481405"}
Jul 01 18:38:28.465886 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.465864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk" event={"ID":"a836d8e1-cc91-4cf3-ada8-b1b5955b63dc","Type":"ContainerStarted","Data":"c9a2a96d6dd580f0016802834a83bf62938335f79cee32de25aa7986fa786a79"}
Jul 01 18:38:28.476090 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.476045    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"]
Jul 01 18:38:28.481669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.481649    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgrd\" (UniqueName: \"kubernetes.io/projected/101b5c81-fe81-4627-9485-a46d1c8844b1-kube-api-access-plgrd\") pod \"console-operator-7fdbdf9c99-jnztt\" (UID: \"101b5c81-fe81-4627-9485-a46d1c8844b1\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.526916 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.526875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca4e42f-afd3-466e-9b65-8958c18336a2-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.527075 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.526928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ca4e42f-afd3-466e-9b65-8958c18336a2-config\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.527075 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.527005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrr9\" (UniqueName: \"kubernetes.io/projected/4ca4e42f-afd3-466e-9b65-8958c18336a2-kube-api-access-dzrr9\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.527824 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.527786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ca4e42f-afd3-466e-9b65-8958c18336a2-config\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.530542 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.530497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ca4e42f-afd3-466e-9b65-8958c18336a2-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.544631 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.544609    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c69kt"]
Jul 01 18:38:28.553292 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.553229    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrr9\" (UniqueName: \"kubernetes.io/projected/4ca4e42f-afd3-466e-9b65-8958c18336a2-kube-api-access-dzrr9\") pod \"service-ca-operator-5d7b7dc95c-p8vps\" (UID: \"4ca4e42f-afd3-466e-9b65-8958c18336a2\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.608237 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.608206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:28.712768 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.712659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"
Jul 01 18:38:28.746767 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:28.746730    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11574496_4380_4df8_a38b_5e732c84d264.slice/crio-d2665b1a8e126e3ef47ef882fb143dc7e0cb64cac309f00f431f7e3495c3d2c0 WatchSource:0}: Error finding container d2665b1a8e126e3ef47ef882fb143dc7e0cb64cac309f00f431f7e3495c3d2c0: Status 404 returned error can't find the container with id d2665b1a8e126e3ef47ef882fb143dc7e0cb64cac309f00f431f7e3495c3d2c0
Jul 01 18:38:28.936460 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.936434    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-jnztt"]
Jul 01 18:38:28.945163 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:28.945132    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101b5c81_fe81_4627_9485_a46d1c8844b1.slice/crio-c968085d7749a28ac800be3f623b4a20d07450c64f61284ca235fece23842939 WatchSource:0}: Error finding container c968085d7749a28ac800be3f623b4a20d07450c64f61284ca235fece23842939: Status 404 returned error can't find the container with id c968085d7749a28ac800be3f623b4a20d07450c64f61284ca235fece23842939
Jul 01 18:38:28.980056 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:28.979994    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps"]
Jul 01 18:38:28.994620 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:28.994594    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca4e42f_afd3_466e_9b65_8958c18336a2.slice/crio-be7aa809a86c214188187720c7087cb638f8e03d4fe810b861e04700916a786e WatchSource:0}: Error finding container be7aa809a86c214188187720c7087cb638f8e03d4fe810b861e04700916a786e: Status 404 returned error can't find the container with id be7aa809a86c214188187720c7087cb638f8e03d4fe810b861e04700916a786e
Jul 01 18:38:29.030752 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.030723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:29.030886 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:29.030860    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:29.030969 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:29.030932    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls podName:b5ee0e9e-2392-45b1-af1f-772e235cafe7 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:31.030911341 +0000 UTC m=+69.265865598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-bjs8g" (UID: "b5ee0e9e-2392-45b1-af1f-772e235cafe7") : secret "samples-operator-tls" not found
Jul 01 18:38:29.472742 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.472703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk" event={"ID":"a836d8e1-cc91-4cf3-ada8-b1b5955b63dc","Type":"ContainerStarted","Data":"d366f37231f392c3e54f7ce4e44d146401c05770f34fdf9383addec17c37fa16"}
Jul 01 18:38:29.474567 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.474517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps" event={"ID":"4ca4e42f-afd3-466e-9b65-8958c18336a2","Type":"ContainerStarted","Data":"be7aa809a86c214188187720c7087cb638f8e03d4fe810b861e04700916a786e"}
Jul 01 18:38:29.476163 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.476115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c69kt" event={"ID":"11574496-4380-4df8-a38b-5e732c84d264","Type":"ContainerStarted","Data":"d2665b1a8e126e3ef47ef882fb143dc7e0cb64cac309f00f431f7e3495c3d2c0"}
Jul 01 18:38:29.477813 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.477787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" event={"ID":"101b5c81-fe81-4627-9485-a46d1c8844b1","Type":"ContainerStarted","Data":"c968085d7749a28ac800be3f623b4a20d07450c64f61284ca235fece23842939"}
Jul 01 18:38:29.505764 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.505506    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-cmldk" podStartSLOduration=1.265661401 podStartE2EDuration="2.505487358s" totalImagesPullingTime="1.239825957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:29.504200787 +0000 UTC m=+67.739155055" watchObservedRunningTime="2026-07-01 18:38:29.505487358 +0000 UTC m=+67.740441663"
Jul 01 18:38:29.635875 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.635835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:38:29.636057 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:29.635992    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 18:38:29.636057 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:29.636058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls podName:6b9359a6-a671-416d-89cd-fc18afd1fdc8 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:01.636039994 +0000 UTC m=+99.870994254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls") pod "dns-default-9s8cn" (UID: "6b9359a6-a671-416d-89cd-fc18afd1fdc8") : secret "dns-default-metrics-tls" not found
Jul 01 18:38:29.720608 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.720537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:29.875644 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:29.875525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:30.137184 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.137116    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7949f867cb-75prg"]
Jul 01 18:38:30.149309 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.149285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.152030 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.152006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 18:38:30.154794 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.154009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 18:38:30.154794 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.154218    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 18:38:30.154794 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.154432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kw6vd\""
Jul 01 18:38:30.162397 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.160833    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 18:38:30.171722 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.171703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7949f867cb-75prg"]
Jul 01 18:38:30.241116 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-bound-sa-token\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-image-registry-private-configuration\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-certificates\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjd6h\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-kube-api-access-xjd6h\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241401 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-ca-trust-extracted\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241457 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241457 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-trusted-ca\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.241557 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.241457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-installation-pull-secrets\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-ca-trust-extracted\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-trusted-ca\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-installation-pull-secrets\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-bound-sa-token\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-image-registry-private-configuration\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-certificates\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.342556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjd6h\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-kube-api-access-xjd6h\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.343366    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-ca-trust-extracted\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.343649    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.343669    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7949f867cb-75prg: secret "image-registry-tls" not found
Jul 01 18:38:30.343838 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.343730    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls podName:f2fe941b-22a0-4cea-90c9-d33f2fe11c7d nodeName:}" failed. No retries permitted until 2026-07-01 18:38:30.843709198 +0000 UTC m=+69.078663465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls") pod "image-registry-7949f867cb-75prg" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d") : secret "image-registry-tls" not found
Jul 01 18:38:30.344572 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.344424    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-certificates\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.345301 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.345258    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-trusted-ca\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.347331 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.347291    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-installation-pull-secrets\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.353573 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.353545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-image-registry-private-configuration\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.359642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.359597    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjd6h\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-kube-api-access-xjd6h\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.362020 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.361997    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-bound-sa-token\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.443880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.443772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:30.444035 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.443963    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:30.444035 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.443967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:30.444035 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.444022    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 18:38:30.444202 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.444039    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:38.444018067 +0000 UTC m=+76.678972317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:30.444202 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.444091    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:38.444081286 +0000 UTC m=+76.679035529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : configmap "kube-rbac-proxy" not found
Jul 01 18:38:30.848008 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:30.847965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:30.848469 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.848141    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:30.848469 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.848158    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7949f867cb-75prg: secret "image-registry-tls" not found
Jul 01 18:38:30.848469 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:30.848232    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls podName:f2fe941b-22a0-4cea-90c9-d33f2fe11c7d nodeName:}" failed. No retries permitted until 2026-07-01 18:38:31.848210393 +0000 UTC m=+70.083164651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls") pod "image-registry-7949f867cb-75prg" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d") : secret "image-registry-tls" not found
Jul 01 18:38:31.049579 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:31.049540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:31.049753 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:31.049692    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:31.049844 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:31.049754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls podName:b5ee0e9e-2392-45b1-af1f-772e235cafe7 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:35.049739134 +0000 UTC m=+73.284693377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-bjs8g" (UID: "b5ee0e9e-2392-45b1-af1f-772e235cafe7") : secret "samples-operator-tls" not found
Jul 01 18:38:31.856727 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:31.856693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:31.857120 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:31.856860    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:31.857120 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:31.856879    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7949f867cb-75prg: secret "image-registry-tls" not found
Jul 01 18:38:31.857120 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:31.856930    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls podName:f2fe941b-22a0-4cea-90c9-d33f2fe11c7d nodeName:}" failed. No retries permitted until 2026-07-01 18:38:33.856915982 +0000 UTC m=+72.091870232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls") pod "image-registry-7949f867cb-75prg" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d") : secret "image-registry-tls" not found
Jul 01 18:38:32.237190 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:32.237157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:32.508312 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:32.508222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:32.853550 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:32.853468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:33.874173 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:33.874139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:33.874655 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:33.874255    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:33.874655 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:33.874267    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7949f867cb-75prg: secret "image-registry-tls" not found
Jul 01 18:38:33.874655 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:33.874319    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls podName:f2fe941b-22a0-4cea-90c9-d33f2fe11c7d nodeName:}" failed. No retries permitted until 2026-07-01 18:38:37.874303799 +0000 UTC m=+76.109258043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls") pod "image-registry-7949f867cb-75prg" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d") : secret "image-registry-tls" not found
Jul 01 18:38:34.176965 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.176939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:34.177101 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:34.177083    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:34.177176 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:34.177166    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls podName:728e3048-f973-425d-9c5f-27a99f6295b5 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:50.177151363 +0000 UTC m=+88.412105619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-5dvqm" (UID: "728e3048-f973-425d-9c5f-27a99f6295b5") : secret "cluster-monitoring-operator-tls" not found
Jul 01 18:38:34.495384 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.495353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/0.log"
Jul 01 18:38:34.495523 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.495403    2577 generic.go:356] "Generic (PLEG): container finished" podID="101b5c81-fe81-4627-9485-a46d1c8844b1" containerID="560cd3557e9131a5993a05b545e0dc626a7a97f9eb399a625adef887daa80af2" exitCode=255
Jul 01 18:38:34.495523 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.495487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" event={"ID":"101b5c81-fe81-4627-9485-a46d1c8844b1","Type":"ContainerDied","Data":"560cd3557e9131a5993a05b545e0dc626a7a97f9eb399a625adef887daa80af2"}
Jul 01 18:38:34.495695 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.495680    2577 scope.go:117] "RemoveContainer" containerID="560cd3557e9131a5993a05b545e0dc626a7a97f9eb399a625adef887daa80af2"
Jul 01 18:38:34.497162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.497130    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k499p" event={"ID":"cba52558-8c44-4d21-94cf-ed28570c6337","Type":"ContainerStarted","Data":"f3bcc7c1740f9bcfe1f36e64bdd2df6230a1e1aadbc28f2f17d0e2ffdb12595f"}
Jul 01 18:38:34.498692 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.498666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps" event={"ID":"4ca4e42f-afd3-466e-9b65-8958c18336a2","Type":"ContainerStarted","Data":"f9a0734a8eb36c4652e33d25f2386c1ccf5256886ba6512d6a43a9dbc3f4a1ec"}
Jul 01 18:38:34.500193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.500171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c69kt" event={"ID":"11574496-4380-4df8-a38b-5e732c84d264","Type":"ContainerStarted","Data":"b1eeca2bfd960175cc3892fe2bef5c866a34733bc29a09fefe4a6c2386b78f9e"}
Jul 01 18:38:34.500330 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.500316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:38:34.661161 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.661112    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps" podStartSLOduration=1.470152389 podStartE2EDuration="6.661095002s" totalImagesPullingTime="5.190942613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:34.59547389 +0000 UTC m=+72.830428156" watchObservedRunningTime="2026-07-01 18:38:34.661095002 +0000 UTC m=+72.896049306"
Jul 01 18:38:34.692623 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.692575    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k499p" podStartSLOduration=66.654506605 podStartE2EDuration="1m12.692553094s" totalImagesPullingTime="6.038046489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:34.662346776 +0000 UTC m=+72.897301041" watchObservedRunningTime="2026-07-01 18:38:34.692553094 +0000 UTC m=+72.927507361"
Jul 01 18:38:34.926046 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:34.925947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:35.084005 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.083972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:35.084180 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:35.084156    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 18:38:35.084254 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:35.084244    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls podName:b5ee0e9e-2392-45b1-af1f-772e235cafe7 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:43.08421988 +0000 UTC m=+81.319174128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-bjs8g" (UID: "b5ee0e9e-2392-45b1-af1f-772e235cafe7") : secret "samples-operator-tls" not found
Jul 01 18:38:35.149880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.149851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:35.505419 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.505396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/1.log"
Jul 01 18:38:35.505890 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.505874    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/0.log"
Jul 01 18:38:35.505961 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.505914    2577 generic.go:356] "Generic (PLEG): container finished" podID="101b5c81-fe81-4627-9485-a46d1c8844b1" containerID="d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443" exitCode=255
Jul 01 18:38:35.506021 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.506005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" event={"ID":"101b5c81-fe81-4627-9485-a46d1c8844b1","Type":"ContainerDied","Data":"d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443"}
Jul 01 18:38:35.506669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.506653    2577 scope.go:117] "RemoveContainer" containerID="560cd3557e9131a5993a05b545e0dc626a7a97f9eb399a625adef887daa80af2"
Jul 01 18:38:35.506760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.506747    2577 scope.go:117] "RemoveContainer" containerID="d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443"
Jul 01 18:38:35.506940 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:35.506910    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-jnztt_openshift-console-operator(101b5c81-fe81-4627-9485-a46d1c8844b1)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podUID="101b5c81-fe81-4627-9485-a46d1c8844b1"
Jul 01 18:38:35.554349 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.554298    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c69kt" podStartSLOduration=68.106347853 podStartE2EDuration="1m13.55427945s" totalImagesPullingTime="5.447931597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:34.693237878 +0000 UTC m=+72.928192167" watchObservedRunningTime="2026-07-01 18:38:35.55427945 +0000 UTC m=+73.789233696"
Jul 01 18:38:35.827654 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:35.827557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.014376 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:36.014342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.277800 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:36.277769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.459905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:36.459870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:36.513725 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:36.513697    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/1.log"
Jul 01 18:38:36.514007 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:36.513993    2577 scope.go:117] "RemoveContainer" containerID="d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443"
Jul 01 18:38:36.514160 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:36.514143    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-jnztt_openshift-console-operator(101b5c81-fe81-4627-9485-a46d1c8844b1)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podUID="101b5c81-fe81-4627-9485-a46d1c8844b1"
Jul 01 18:38:36.748483 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:36.748446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:37.342367 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:37.342333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:37.607348 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:37.607273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:37.907053 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:37.906971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:37.907192 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:37.907082    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:37.907192 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:37.907094    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7949f867cb-75prg: secret "image-registry-tls" not found
Jul 01 18:38:37.907192 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:37.907152    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls podName:f2fe941b-22a0-4cea-90c9-d33f2fe11c7d nodeName:}" failed. No retries permitted until 2026-07-01 18:38:45.90713755 +0000 UTC m=+84.142091799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls") pod "image-registry-7949f867cb-75prg" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d") : secret "image-registry-tls" not found
Jul 01 18:38:38.260249 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.260221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:38.511306 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.511230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:38.511652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.511477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:38.511652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.511564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:38.511726 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:38.511676    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:38.511755 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:38.511736    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls podName:375d8359-feea-4c7a-b45e-169f4badc3e4 nodeName:}" failed. No retries permitted until 2026-07-01 18:38:54.511721442 +0000 UTC m=+92.746675879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gpcfz" (UID: "375d8359-feea-4c7a-b45e-169f4badc3e4") : secret "insights-runtime-extractor-tls" not found
Jul 01 18:38:38.511956 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.511940    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/375d8359-feea-4c7a-b45e-169f4badc3e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:38.608846 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.608818    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:38.609008 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.608867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:38.609171 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:38.609160    2577 scope.go:117] "RemoveContainer" containerID="d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443"
Jul 01 18:38:38.609311 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:38.609296    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-jnztt_openshift-console-operator(101b5c81-fe81-4627-9485-a46d1c8844b1)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podUID="101b5c81-fe81-4627-9485-a46d1c8844b1"
Jul 01 18:38:39.556600 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:39.556571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:41.129457 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:41.129427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:42.689336 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:42.689304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:42.971118 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:42.971088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:43.147885 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.147848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:43.150294 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.150269    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b5ee0e9e-2392-45b1-af1f-772e235cafe7-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-bjs8g\" (UID: \"b5ee0e9e-2392-45b1-af1f-772e235cafe7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:43.193274 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.193243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:43.277454 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.277376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"
Jul 01 18:38:43.411228 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.411193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:43.450213 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.450148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g"]
Jul 01 18:38:43.535760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.535676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g" event={"ID":"b5ee0e9e-2392-45b1-af1f-772e235cafe7","Type":"ContainerStarted","Data":"0f9350a1afd868e89bc3670547fe9a232ab34424ae17b0484aa32b0b3ee12f65"}
Jul 01 18:38:43.786694 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:43.786618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:44.443700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:44.443663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:44.972722 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:44.972693    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:45.253335 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.253305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:45.545145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.545063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g" event={"ID":"b5ee0e9e-2392-45b1-af1f-772e235cafe7","Type":"ContainerStarted","Data":"5187719d66b58f00145e4dc371619352a3c599ca4d65d2649e17963b4eb5f665"}
Jul 01 18:38:45.545145 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.545116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g" event={"ID":"b5ee0e9e-2392-45b1-af1f-772e235cafe7","Type":"ContainerStarted","Data":"747c3415b09d6b2708312ef0574a0a8645f64a71bdd30a32aaf282d0e00609a5"}
Jul 01 18:38:45.576636 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.576607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:45.599080 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.596997    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-bjs8g" podStartSLOduration=17.134618381 podStartE2EDuration="18.596978884s" totalImagesPullingTime="1.462360503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:45.596092716 +0000 UTC m=+83.831046983" watchObservedRunningTime="2026-07-01 18:38:45.596978884 +0000 UTC m=+83.831933151"
Jul 01 18:38:45.792915 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.792884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:45.973928 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:45.973895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:38:45.974299 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:45.974039    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 18:38:45.974299 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:45.974057    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7949f867cb-75prg: secret "image-registry-tls" not found
Jul 01 18:38:45.974299 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:45.974113    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls podName:f2fe941b-22a0-4cea-90c9-d33f2fe11c7d nodeName:}" failed. No retries permitted until 2026-07-01 18:39:01.974096218 +0000 UTC m=+100.209050462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls") pod "image-registry-7949f867cb-75prg" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d") : secret "image-registry-tls" not found
Jul 01 18:38:46.168462 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:46.168423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:46.606586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:46.606554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:46.861103 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:46.861030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:47.005592 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:47.005563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:47.189397 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:47.189315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:47.349193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:47.349162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:47.581693 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:47.581662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:47.872449 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:47.872368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:48.410640 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:48.410613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:48.968159 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:48.968129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.236475 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:49.236450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.443348 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:49.443317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.623254 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:49.623174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:49.819187 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:49.819153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.204375 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.204343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:50.206917 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.206892    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/728e3048-f973-425d-9c5f-27a99f6295b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-5dvqm\" (UID: \"728e3048-f973-425d-9c5f-27a99f6295b5\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:50.331390 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.331356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:50.432994 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.432965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m76ht\""
Jul 01 18:38:50.437173 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.437158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"
Jul 01 18:38:50.598039 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.598013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm"]
Jul 01 18:38:50.602867 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:50.602799    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728e3048_f973_425d_9c5f_27a99f6295b5.slice/crio-853ec97417804d589fca57a5afc0f9679ac1a3ebd9f7613358926b8f7e97bdb0 WatchSource:0}: Error finding container 853ec97417804d589fca57a5afc0f9679ac1a3ebd9f7613358926b8f7e97bdb0: Status 404 returned error can't find the container with id 853ec97417804d589fca57a5afc0f9679ac1a3ebd9f7613358926b8f7e97bdb0
Jul 01 18:38:50.891003 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:50.890916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:51.408622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:51.408586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:51.564089 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:51.564053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm" event={"ID":"728e3048-f973-425d-9c5f-27a99f6295b5","Type":"ContainerStarted","Data":"853ec97417804d589fca57a5afc0f9679ac1a3ebd9f7613358926b8f7e97bdb0"}
Jul 01 18:38:51.681191 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:51.681117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:51.983244 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:51.983200    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:52.205022 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.204994    2577 scope.go:117] "RemoveContainer" containerID="d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443"
Jul 01 18:38:52.228156 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.228134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:52.568634 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.568557    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:38:52.569021 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.569007    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/1.log"
Jul 01 18:38:52.569086 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.569041    2577 generic.go:356] "Generic (PLEG): container finished" podID="101b5c81-fe81-4627-9485-a46d1c8844b1" containerID="77a209a83fc1a3fd99a25291d4f8e6dc4c2e670aa257e89031b725dc9a65b1b1" exitCode=255
Jul 01 18:38:52.569129 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.569112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" event={"ID":"101b5c81-fe81-4627-9485-a46d1c8844b1","Type":"ContainerDied","Data":"77a209a83fc1a3fd99a25291d4f8e6dc4c2e670aa257e89031b725dc9a65b1b1"}
Jul 01 18:38:52.569181 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.569168    2577 scope.go:117] "RemoveContainer" containerID="d30efaf2bd88fb5146cc9714af2071f0f8e6d4b608cc975a48857053fb334443"
Jul 01 18:38:52.569510 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.569488    2577 scope.go:117] "RemoveContainer" containerID="77a209a83fc1a3fd99a25291d4f8e6dc4c2e670aa257e89031b725dc9a65b1b1"
Jul 01 18:38:52.569697 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:52.569678    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-jnztt_openshift-console-operator(101b5c81-fe81-4627-9485-a46d1c8844b1)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podUID="101b5c81-fe81-4627-9485-a46d1c8844b1"
Jul 01 18:38:52.570729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.570700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm" event={"ID":"728e3048-f973-425d-9c5f-27a99f6295b5","Type":"ContainerStarted","Data":"80ea3eca6771e6939874522cad7da596566d7d39334739042bd8f7ce6dc78bde"}
Jul 01 18:38:52.572937 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.572918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:52.714234 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.714187    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-5dvqm" podStartSLOduration=33.133324817 podStartE2EDuration="34.714169309s" totalImagesPullingTime="1.580844492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:38:52.713634346 +0000 UTC m=+90.948588613" watchObservedRunningTime="2026-07-01 18:38:52.714169309 +0000 UTC m=+90.949123575"
Jul 01 18:38:52.849823 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:52.849729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:53.035942 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:53.035913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:53.236527 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:53.236495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:53.565297 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:53.565215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:38:53.574771 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:53.574750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:38:54.536610 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:54.536567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:54.538774 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:54.538755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/375d8359-feea-4c7a-b45e-169f4badc3e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpcfz\" (UID: \"375d8359-feea-4c7a-b45e-169f4badc3e4\") " pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:54.804973 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:54.804892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gpcfz"
Jul 01 18:38:54.976388 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:54.976360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gpcfz"]
Jul 01 18:38:54.979624 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:38:54.979595    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375d8359_feea_4c7a_b45e_169f4badc3e4.slice/crio-92c72d0f06f02094a6059df3aa340116788039b06aa6f9a457e8f793dcbcb59b WatchSource:0}: Error finding container 92c72d0f06f02094a6059df3aa340116788039b06aa6f9a457e8f793dcbcb59b: Status 404 returned error can't find the container with id 92c72d0f06f02094a6059df3aa340116788039b06aa6f9a457e8f793dcbcb59b
Jul 01 18:38:55.583642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:55.583518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpcfz" event={"ID":"375d8359-feea-4c7a-b45e-169f4badc3e4","Type":"ContainerStarted","Data":"92c72d0f06f02094a6059df3aa340116788039b06aa6f9a457e8f793dcbcb59b"}
Jul 01 18:38:58.608880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:58.608846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:58.609423 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:58.608910    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:38:58.609423 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:58.609293    2577 scope.go:117] "RemoveContainer" containerID="77a209a83fc1a3fd99a25291d4f8e6dc4c2e670aa257e89031b725dc9a65b1b1"
Jul 01 18:38:58.609546 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:38:58.609515    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-jnztt_openshift-console-operator(101b5c81-fe81-4627-9485-a46d1c8844b1)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podUID="101b5c81-fe81-4627-9485-a46d1c8844b1"
Jul 01 18:38:59.599854 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:59.599757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpcfz" event={"ID":"375d8359-feea-4c7a-b45e-169f4badc3e4","Type":"ContainerStarted","Data":"1bc521a2cbe510710a2a76a533628fd5dffecbf6e646361a1e6f1d906c351f76"}
Jul 01 18:38:59.599854 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:38:59.599824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpcfz" event={"ID":"375d8359-feea-4c7a-b45e-169f4badc3e4","Type":"ContainerStarted","Data":"292d56efa4c5cdc1850d0693f2c8eb5eb26bbadda67cb94cbe32eca5a4719182"}
Jul 01 18:39:01.699859 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:01.699821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:39:01.701911 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:01.701885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6b9359a6-a671-416d-89cd-fc18afd1fdc8-metrics-tls\") pod \"dns-default-9s8cn\" (UID: \"6b9359a6-a671-416d-89cd-fc18afd1fdc8\") " pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:39:01.949008 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:01.948978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4nfs\""
Jul 01 18:39:01.956084 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:01.956030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:39:02.002478 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.002445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:02.004693 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.004669    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"image-registry-7949f867cb-75prg\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") " pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:02.111281 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.111230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9s8cn"]
Jul 01 18:39:02.114507 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:02.114486    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9359a6_a671_416d_89cd_fc18afd1fdc8.slice/crio-8723fdb44fb741b3a223fea8dda2470de8fe893b5ad5140ca2202eea811faf09 WatchSource:0}: Error finding container 8723fdb44fb741b3a223fea8dda2470de8fe893b5ad5140ca2202eea811faf09: Status 404 returned error can't find the container with id 8723fdb44fb741b3a223fea8dda2470de8fe893b5ad5140ca2202eea811faf09
Jul 01 18:39:02.264060 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.264031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:02.409167 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.409136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7949f867cb-75prg"]
Jul 01 18:39:02.416855 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:02.416834    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe941b_22a0_4cea_90c9_d33f2fe11c7d.slice/crio-535469dffd27979dfc8d71487e6be303899a0d15c773c30f775da61fb6314835 WatchSource:0}: Error finding container 535469dffd27979dfc8d71487e6be303899a0d15c773c30f775da61fb6314835: Status 404 returned error can't find the container with id 535469dffd27979dfc8d71487e6be303899a0d15c773c30f775da61fb6314835
Jul 01 18:39:02.612166 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.612068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7949f867cb-75prg" event={"ID":"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d","Type":"ContainerStarted","Data":"3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79"}
Jul 01 18:39:02.612166 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.612139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:02.612166 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.612163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7949f867cb-75prg" event={"ID":"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d","Type":"ContainerStarted","Data":"535469dffd27979dfc8d71487e6be303899a0d15c773c30f775da61fb6314835"}
Jul 01 18:39:02.614478 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.614442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpcfz" event={"ID":"375d8359-feea-4c7a-b45e-169f4badc3e4","Type":"ContainerStarted","Data":"645e5c8ec5dfd0e115fa3072c72b865a5dc49b1498ea3e475f447a7a3e47f0e7"}
Jul 01 18:39:02.615677 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.615652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9s8cn" event={"ID":"6b9359a6-a671-416d-89cd-fc18afd1fdc8","Type":"ContainerStarted","Data":"8723fdb44fb741b3a223fea8dda2470de8fe893b5ad5140ca2202eea811faf09"}
Jul 01 18:39:02.684159 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.684105    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gpcfz" podStartSLOduration=34.045142227 podStartE2EDuration="40.684086122s" totalImagesPullingTime="6.638943895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 18:38:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:02.683301615 +0000 UTC m=+100.918255904" watchObservedRunningTime="2026-07-01 18:39:02.684086122 +0000 UTC m=+100.919040388"
Jul 01 18:39:02.685096 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:02.685053    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7949f867cb-75prg" podStartSLOduration=32.685040087 podStartE2EDuration="32.685040087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:02.64615934 +0000 UTC m=+100.881113630" watchObservedRunningTime="2026-07-01 18:39:02.685040087 +0000 UTC m=+100.919994357"
Jul 01 18:39:04.624716 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:04.624628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9s8cn" event={"ID":"6b9359a6-a671-416d-89cd-fc18afd1fdc8","Type":"ContainerStarted","Data":"b8441718bb672ba0904ae16ca566289dd8a25313a70a03ff561c694304d6d925"}
Jul 01 18:39:04.624716 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:04.624681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9s8cn" event={"ID":"6b9359a6-a671-416d-89cd-fc18afd1fdc8","Type":"ContainerStarted","Data":"446f0e1f008f886ed9fbd5c7b5116760a69863e7edfae948a1e06ce3503d2d96"}
Jul 01 18:39:04.625128 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:04.624753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:39:04.652357 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:04.652306    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9s8cn" podStartSLOduration=65.714001196 podStartE2EDuration="1m7.652289735s" totalImagesPullingTime="1.938288539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:04.651943478 +0000 UTC m=+102.886897795" watchObservedRunningTime="2026-07-01 18:39:04.652289735 +0000 UTC m=+102.887244003"
Jul 01 18:39:05.508835 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:05.508792    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c69kt"
Jul 01 18:39:11.202907 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:11.202876    2577 scope.go:117] "RemoveContainer" containerID="77a209a83fc1a3fd99a25291d4f8e6dc4c2e670aa257e89031b725dc9a65b1b1"
Jul 01 18:39:11.203368 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:39:11.203039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-jnztt_openshift-console-operator(101b5c81-fe81-4627-9485-a46d1c8844b1)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podUID="101b5c81-fe81-4627-9485-a46d1c8844b1"
Jul 01 18:39:12.413358 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.413330    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7949f867cb-75prg"]
Jul 01 18:39:12.426771 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.426740    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"]
Jul 01 18:39:12.428740 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.428725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.435696 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.435672    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 18:39:12.435800 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.435767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d94rr\""
Jul 01 18:39:12.446247 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.446228    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 18:39:12.461503 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.461480    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"]
Jul 01 18:39:12.525311 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.525283    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"]
Jul 01 18:39:12.527150 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.527136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:12.531789 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.531767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-46n29\""
Jul 01 18:39:12.532047 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.532032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 01 18:39:12.580642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.580614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a97dee06-2aa7-476b-9ca1-03755b87c713-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-cw2s5\" (UID: \"a97dee06-2aa7-476b-9ca1-03755b87c713\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.580768 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.580656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a97dee06-2aa7-476b-9ca1-03755b87c713-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-cw2s5\" (UID: \"a97dee06-2aa7-476b-9ca1-03755b87c713\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.581461 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.581443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:12.587995 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.587975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"]
Jul 01 18:39:12.681550 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.681479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a97dee06-2aa7-476b-9ca1-03755b87c713-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-cw2s5\" (UID: \"a97dee06-2aa7-476b-9ca1-03755b87c713\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.681550 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.681513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3fa3bbc-7729-46bc-b759-0b863aed1c22-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-k6zwt\" (UID: \"b3fa3bbc-7729-46bc-b759-0b863aed1c22\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:12.681550 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.681536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a97dee06-2aa7-476b-9ca1-03755b87c713-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-cw2s5\" (UID: \"a97dee06-2aa7-476b-9ca1-03755b87c713\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.682206 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.682170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a97dee06-2aa7-476b-9ca1-03755b87c713-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-cw2s5\" (UID: \"a97dee06-2aa7-476b-9ca1-03755b87c713\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.683937 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.683914    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a97dee06-2aa7-476b-9ca1-03755b87c713-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-cw2s5\" (UID: \"a97dee06-2aa7-476b-9ca1-03755b87c713\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.737994 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.737958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"
Jul 01 18:39:12.782981 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.782718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3fa3bbc-7729-46bc-b759-0b863aed1c22-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-k6zwt\" (UID: \"b3fa3bbc-7729-46bc-b759-0b863aed1c22\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:12.785548 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.785482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3fa3bbc-7729-46bc-b759-0b863aed1c22-tls-certificates\") pod \"prometheus-operator-admission-webhook-7ddc8784d4-k6zwt\" (UID: \"b3fa3bbc-7729-46bc-b759-0b863aed1c22\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:12.799747 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.799723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:12.835161 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.835129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:12.905940 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.905916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5"]
Jul 01 18:39:12.907857 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:12.907831    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97dee06_2aa7_476b_9ca1_03755b87c713.slice/crio-acabd02f63344383c6cc744ec4b8309487f35c839d5a29c2c3811f897c3bbe86 WatchSource:0}: Error finding container acabd02f63344383c6cc744ec4b8309487f35c839d5a29c2c3811f897c3bbe86: Status 404 returned error can't find the container with id acabd02f63344383c6cc744ec4b8309487f35c839d5a29c2c3811f897c3bbe86
Jul 01 18:39:12.985264 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:12.985235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"]
Jul 01 18:39:12.985551 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:12.985532    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fa3bbc_7729_46bc_b759_0b863aed1c22.slice/crio-c77d26c6fd39637a32170983294faf108a5827adf6ff879f817d735846e46bd5 WatchSource:0}: Error finding container c77d26c6fd39637a32170983294faf108a5827adf6ff879f817d735846e46bd5: Status 404 returned error can't find the container with id c77d26c6fd39637a32170983294faf108a5827adf6ff879f817d735846e46bd5
Jul 01 18:39:13.025052 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:13.025027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:13.656988 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:13.656947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5" event={"ID":"a97dee06-2aa7-476b-9ca1-03755b87c713","Type":"ContainerStarted","Data":"acabd02f63344383c6cc744ec4b8309487f35c839d5a29c2c3811f897c3bbe86"}
Jul 01 18:39:13.658183 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:13.658152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt" event={"ID":"b3fa3bbc-7729-46bc-b759-0b863aed1c22","Type":"ContainerStarted","Data":"c77d26c6fd39637a32170983294faf108a5827adf6ff879f817d735846e46bd5"}
Jul 01 18:39:14.630994 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.630912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9s8cn"
Jul 01 18:39:14.663316 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.663283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5" event={"ID":"a97dee06-2aa7-476b-9ca1-03755b87c713","Type":"ContainerStarted","Data":"0b7566856f03968354c944151bb4d79138a0401660612bb3c545f1f9b19c9389"}
Jul 01 18:39:14.665003 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.664980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt" event={"ID":"b3fa3bbc-7729-46bc-b759-0b863aed1c22","Type":"ContainerStarted","Data":"f369a79d971f47804bdb8e1936b158fcdd3ac8b771c04b4f75138689b07229c7"}
Jul 01 18:39:14.665183 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.665170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:14.669414 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.669396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt"
Jul 01 18:39:14.701467 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.701424    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-cw2s5" podStartSLOduration=1.327485823 podStartE2EDuration="2.701407768s" totalImagesPullingTime="1.373921945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:14.700839221 +0000 UTC m=+112.935793502" watchObservedRunningTime="2026-07-01 18:39:14.701407768 +0000 UTC m=+112.936362034"
Jul 01 18:39:14.958323 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:14.958242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:15.064142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.064113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:15.213575 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.213502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:15.328939 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.328892    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7ddc8784d4-k6zwt" podStartSLOduration=2.033045556 podStartE2EDuration="3.328876019s" totalImagesPullingTime="1.295830463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:14.751285455 +0000 UTC m=+112.986239722" watchObservedRunningTime="2026-07-01 18:39:15.328876019 +0000 UTC m=+113.563830283"
Jul 01 18:39:15.329951 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.329930    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"]
Jul 01 18:39:15.332143 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.332126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.334570 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.334552    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h542x\""
Jul 01 18:39:15.335400 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.335380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 18:39:15.335504 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.335400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 18:39:15.335669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.335655    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 18:39:15.338770 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.338746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:15.364114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.364087    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"]
Jul 01 18:39:15.504991 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.504963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a0f5be-9501-40d5-af2e-e298213cf9e6-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.505147 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.504997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20a0f5be-9501-40d5-af2e-e298213cf9e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.505147 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.505042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmh2l\" (UniqueName: \"kubernetes.io/projected/20a0f5be-9501-40d5-af2e-e298213cf9e6-kube-api-access-qmh2l\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.505147 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.505117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a0f5be-9501-40d5-af2e-e298213cf9e6-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.606225 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.606194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a0f5be-9501-40d5-af2e-e298213cf9e6-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.606225 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.606229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20a0f5be-9501-40d5-af2e-e298213cf9e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.606448 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.606285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmh2l\" (UniqueName: \"kubernetes.io/projected/20a0f5be-9501-40d5-af2e-e298213cf9e6-kube-api-access-qmh2l\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.606448 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.606307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a0f5be-9501-40d5-af2e-e298213cf9e6-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.606900 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.606876    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20a0f5be-9501-40d5-af2e-e298213cf9e6-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.608676 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.608653    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/20a0f5be-9501-40d5-af2e-e298213cf9e6-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.608778 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.608701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20a0f5be-9501-40d5-af2e-e298213cf9e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.638063 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.638032    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmh2l\" (UniqueName: \"kubernetes.io/projected/20a0f5be-9501-40d5-af2e-e298213cf9e6-kube-api-access-qmh2l\") pod \"prometheus-operator-7b7d896497-bk6ch\" (UID: \"20a0f5be-9501-40d5-af2e-e298213cf9e6\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.640878 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.640857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"
Jul 01 18:39:15.783531 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:15.783461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-bk6ch"]
Jul 01 18:39:15.786474 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:15.786448    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a0f5be_9501_40d5_af2e_e298213cf9e6.slice/crio-bef607804f7a0df3930b91ed92686afb8669ed71b45dfe7e5267c3d301e89786 WatchSource:0}: Error finding container bef607804f7a0df3930b91ed92686afb8669ed71b45dfe7e5267c3d301e89786: Status 404 returned error can't find the container with id bef607804f7a0df3930b91ed92686afb8669ed71b45dfe7e5267c3d301e89786
Jul 01 18:39:16.673199 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:16.673166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch" event={"ID":"20a0f5be-9501-40d5-af2e-e298213cf9e6","Type":"ContainerStarted","Data":"bef607804f7a0df3930b91ed92686afb8669ed71b45dfe7e5267c3d301e89786"}
Jul 01 18:39:17.521889 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:17.521858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:17.678213 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:17.678181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch" event={"ID":"20a0f5be-9501-40d5-af2e-e298213cf9e6","Type":"ContainerStarted","Data":"1de24ba6b933dbf9e04bbfe69e613d9eba9aa0e1701d12457091ace955e493b8"}
Jul 01 18:39:17.678372 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:17.678233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch" event={"ID":"20a0f5be-9501-40d5-af2e-e298213cf9e6","Type":"ContainerStarted","Data":"32324c2567fe74657420606bc9d5f64c07667e82beb0cb0ccd0799f287035d33"}
Jul 01 18:39:17.709384 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:17.709342    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-bk6ch" podStartSLOduration=1.416757518 podStartE2EDuration="2.709326037s" totalImagesPullingTime="1.292568519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:17.707387939 +0000 UTC m=+115.942342216" watchObservedRunningTime="2026-07-01 18:39:17.709326037 +0000 UTC m=+115.944280301"
Jul 01 18:39:19.688758 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:19.688727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:19.877790 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:19.877763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:20.199050 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.199020    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:20.379727 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.379701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:20.512816 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.512773    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"]
Jul 01 18:39:20.515945 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.515927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.527924 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.527905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 01 18:39:20.528004 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.527941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v4gzs\""
Jul 01 18:39:20.528083 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.528065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 01 18:39:20.532051 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.532030    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fc6zk"]
Jul 01 18:39:20.534090 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.534076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544619 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.544713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfxm\" (UniqueName: \"kubernetes.io/projected/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-kube-api-access-jrfxm\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.544713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-wtmp\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.544825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-sys\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca8c047d-edd8-453e-ba5f-e2e62c926a23-metrics-client-ca\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4vh\" (UniqueName: \"kubernetes.io/projected/ca8c047d-edd8-453e-ba5f-e2e62c926a23-kube-api-access-pv4vh\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544949 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-root\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544949 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-tls\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544949 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.544949 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-textfile\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.544949 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.544929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.548420 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.548405    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 18:39:20.548497 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.548407    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 18:39:20.548497 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.548456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 18:39:20.548699 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.548685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6hz9l\""
Jul 01 18:39:20.576611 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.576586    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"]
Jul 01 18:39:20.639484 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.639458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:20.645558 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.645673 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-textfile\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.645673 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.645673 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.645673 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfxm\" (UniqueName: \"kubernetes.io/projected/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-kube-api-access-jrfxm\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.645914 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-wtmp\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.645914 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.645914 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-sys\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.645914 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca8c047d-edd8-453e-ba5f-e2e62c926a23-metrics-client-ca\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.645914 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4vh\" (UniqueName: \"kubernetes.io/projected/ca8c047d-edd8-453e-ba5f-e2e62c926a23-kube-api-access-pv4vh\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645932    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-wtmp\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645958    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-textfile\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.645969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-root\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.646013    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-sys\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.646029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca8c047d-edd8-453e-ba5f-e2e62c926a23-root\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.646058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-tls\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646600 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.646572    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.646600 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.646591    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.646760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.646619    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca8c047d-edd8-453e-ba5f-e2e62c926a23-metrics-client-ca\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.648127 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.648109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.648228 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.648131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.648346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.648328    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca8c047d-edd8-453e-ba5f-e2e62c926a23-node-exporter-tls\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.648478 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.648461    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.669010 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.668983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfxm\" (UniqueName: \"kubernetes.io/projected/5ae8c8ef-7358-4be8-8a97-283cda0d6eb7-kube-api-access-jrfxm\") pod \"openshift-state-metrics-57f99c8b49-l74kl\" (UID: \"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.686205 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.686186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4vh\" (UniqueName: \"kubernetes.io/projected/ca8c047d-edd8-453e-ba5f-e2e62c926a23-kube-api-access-pv4vh\") pod \"node-exporter-fc6zk\" (UID: \"ca8c047d-edd8-453e-ba5f-e2e62c926a23\") " pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.816115 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.816039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:20.824987 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.824957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"
Jul 01 18:39:20.841664 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.841643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fc6zk"
Jul 01 18:39:20.851103 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:20.851073    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8c047d_edd8_453e_ba5f_e2e62c926a23.slice/crio-7705df81462de6cda4b547fa553aff9e55601835a5735f84ce02a683235370cb WatchSource:0}: Error finding container 7705df81462de6cda4b547fa553aff9e55601835a5735f84ce02a683235370cb: Status 404 returned error can't find the container with id 7705df81462de6cda4b547fa553aff9e55601835a5735f84ce02a683235370cb
Jul 01 18:39:20.971845 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:20.971797    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl"]
Jul 01 18:39:20.975014 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:20.974990    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae8c8ef_7358_4be8_8a97_283cda0d6eb7.slice/crio-51774f3af9cb9d6ee53a0133524b644a6836dd872c1d4e1f8da796e4086f70a6 WatchSource:0}: Error finding container 51774f3af9cb9d6ee53a0133524b644a6836dd872c1d4e1f8da796e4086f70a6: Status 404 returned error can't find the container with id 51774f3af9cb9d6ee53a0133524b644a6836dd872c1d4e1f8da796e4086f70a6
Jul 01 18:39:21.253368 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.253318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:21.402971 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.402885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:21.547996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.547960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:21.696577 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.696495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6zk" event={"ID":"ca8c047d-edd8-453e-ba5f-e2e62c926a23","Type":"ContainerStarted","Data":"7705df81462de6cda4b547fa553aff9e55601835a5735f84ce02a683235370cb"}
Jul 01 18:39:21.698662 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.698630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl" event={"ID":"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7","Type":"ContainerStarted","Data":"35711d6ae6d74cebc4462f76a70f7a871517ea33badfad7882eca7462a7ad845"}
Jul 01 18:39:21.698793 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.698685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl" event={"ID":"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7","Type":"ContainerStarted","Data":"867d3b05fdb542829cd988c2d8240276ba494977cb4388e6a88ebbd8e126b10b"}
Jul 01 18:39:21.698793 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.698731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl" event={"ID":"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7","Type":"ContainerStarted","Data":"51774f3af9cb9d6ee53a0133524b644a6836dd872c1d4e1f8da796e4086f70a6"}
Jul 01 18:39:21.708907 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:21.708874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:22.025969 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.025929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:22.180581 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.180549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:22.203766 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.203680    2577 scope.go:117] "RemoveContainer" containerID="77a209a83fc1a3fd99a25291d4f8e6dc4c2e670aa257e89031b725dc9a65b1b1"
Jul 01 18:39:22.420416 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.420376    2577 patch_prober.go:28] interesting pod/image-registry-7949f867cb-75prg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 18:39:22.420575 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.420443    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7949f867cb-75prg" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:39:22.703867 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.703831    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca8c047d-edd8-453e-ba5f-e2e62c926a23" containerID="ade8716771e5f0da7051ff2baf7c05b4bd1b5f0eef835e4a6d6fe5d4eda3476b" exitCode=0
Jul 01 18:39:22.704046 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.703918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6zk" event={"ID":"ca8c047d-edd8-453e-ba5f-e2e62c926a23","Type":"ContainerDied","Data":"ade8716771e5f0da7051ff2baf7c05b4bd1b5f0eef835e4a6d6fe5d4eda3476b"}
Jul 01 18:39:22.706171 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.706144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl" event={"ID":"5ae8c8ef-7358-4be8-8a97-283cda0d6eb7","Type":"ContainerStarted","Data":"058709a63045744d8e194ee68a44a2161d67eb9400ba0e335a92b7d093b3be48"}
Jul 01 18:39:22.707900 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.707883    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:39:22.708002 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.707967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" event={"ID":"101b5c81-fe81-4627-9485-a46d1c8844b1","Type":"ContainerStarted","Data":"ce936da2823406a10c6a366d34f0f4502c8629e3175ea71e9ca25e1fd1db1bf8"}
Jul 01 18:39:22.709708 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.709689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:39:22.714457 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.714439    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt"
Jul 01 18:39:22.910296 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.910146    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-jnztt" podStartSLOduration=49.672018474 podStartE2EDuration="54.910123428s" totalImagesPullingTime="5.238104954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:22.835636928 +0000 UTC m=+121.070591211" watchObservedRunningTime="2026-07-01 18:39:22.910123428 +0000 UTC m=+121.145077696"
Jul 01 18:39:22.971580 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.971529    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-l74kl" podStartSLOduration=2.058664692 podStartE2EDuration="2.971510848s" totalImagesPullingTime="912.846156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:22.909420082 +0000 UTC m=+121.144374374" watchObservedRunningTime="2026-07-01 18:39:22.971510848 +0000 UTC m=+121.206465114"
Jul 01 18:39:22.972304 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.972287    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-qkt6x"]
Jul 01 18:39:22.975470 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.975450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:22.978232 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.978213    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-76rjp\""
Jul 01 18:39:22.978713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.978682    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 18:39:22.978904 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:22.978888    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 18:39:23.023498 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.023469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-qkt6x"]
Jul 01 18:39:23.043304 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.043277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:23.065609 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.065580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m896m\" (UniqueName: \"kubernetes.io/projected/f2fe2315-f9aa-426d-9adf-93ff2f0bd73f-kube-api-access-m896m\") pod \"downloads-5499b6fb8f-qkt6x\" (UID: \"f2fe2315-f9aa-426d-9adf-93ff2f0bd73f\") " pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:23.166520 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.166484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m896m\" (UniqueName: \"kubernetes.io/projected/f2fe2315-f9aa-426d-9adf-93ff2f0bd73f-kube-api-access-m896m\") pod \"downloads-5499b6fb8f-qkt6x\" (UID: \"f2fe2315-f9aa-426d-9adf-93ff2f0bd73f\") " pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:23.185445 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.185411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m896m\" (UniqueName: \"kubernetes.io/projected/f2fe2315-f9aa-426d-9adf-93ff2f0bd73f-kube-api-access-m896m\") pod \"downloads-5499b6fb8f-qkt6x\" (UID: \"f2fe2315-f9aa-426d-9adf-93ff2f0bd73f\") " pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:23.199133 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.199103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:23.284683 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.284613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:23.344933 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.344898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:23.441561 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.441538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-qkt6x"]
Jul 01 18:39:23.442932 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:23.442900    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe2315_f9aa_426d_9adf_93ff2f0bd73f.slice/crio-60603ca177a92eb2d1144467b6ab5ba959a39b610b85e3b73b1df0eed384eff7 WatchSource:0}: Error finding container 60603ca177a92eb2d1144467b6ab5ba959a39b610b85e3b73b1df0eed384eff7: Status 404 returned error can't find the container with id 60603ca177a92eb2d1144467b6ab5ba959a39b610b85e3b73b1df0eed384eff7
Jul 01 18:39:23.610108 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.610023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:23.653547 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.653523    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-645f96b865-npct9"]
Jul 01 18:39:23.658350 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.658329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.668076 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 01 18:39:23.668197 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 01 18:39:23.668197 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668138    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 01 18:39:23.668197 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 01 18:39:23.668197 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 01 18:39:23.668641 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qnmj4\""
Jul 01 18:39:23.668890 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.668877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-hfjge5cidd0\""
Jul 01 18:39:23.670104 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670158 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670210 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-tls\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-grpc-tls\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670320 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6d2\" (UniqueName: \"kubernetes.io/projected/cc539bb8-2af5-4c86-83fc-d5eb5046950a-kube-api-access-kb6d2\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670320 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc539bb8-2af5-4c86-83fc-d5eb5046950a-metrics-client-ca\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670398 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.670398 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.670370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.713702 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.713668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6zk" event={"ID":"ca8c047d-edd8-453e-ba5f-e2e62c926a23","Type":"ContainerStarted","Data":"4edf33db0197c30dd71fd181717673a2294365ab6e76d69739e62f952c5f83f1"}
Jul 01 18:39:23.713885 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.713725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc6zk" event={"ID":"ca8c047d-edd8-453e-ba5f-e2e62c926a23","Type":"ContainerStarted","Data":"c92a6b5b203c27fdcad2a0013e3a476ba3c1cb139d24e892ef64804fa608540a"}
Jul 01 18:39:23.714758 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.714734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-qkt6x" event={"ID":"f2fe2315-f9aa-426d-9adf-93ff2f0bd73f","Type":"ContainerStarted","Data":"60603ca177a92eb2d1144467b6ab5ba959a39b610b85e3b73b1df0eed384eff7"}
Jul 01 18:39:23.771696 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.771655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.771696 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.771697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.771965 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.771915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-tls\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.772031 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.771984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-grpc-tls\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.772492 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.772217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6d2\" (UniqueName: \"kubernetes.io/projected/cc539bb8-2af5-4c86-83fc-d5eb5046950a-kube-api-access-kb6d2\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.772492 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.772258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc539bb8-2af5-4c86-83fc-d5eb5046950a-metrics-client-ca\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.772492 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.772310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.772492 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.772364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.773658 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.773085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc539bb8-2af5-4c86-83fc-d5eb5046950a-metrics-client-ca\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.774758 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.774720    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.774936 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.774916    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-tls\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.774998 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.774937    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-grpc-tls\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.775421 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.775400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.775622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.775605    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.776128 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.776106    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc539bb8-2af5-4c86-83fc-d5eb5046950a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.787030 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.786979    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fc6zk" podStartSLOduration=2.850408292 podStartE2EDuration="3.786965702s" totalImagesPullingTime="870.089317ms" totalInitContainerRuntime="66.468093ms" isStatefulPod=true podCreationTimestamp="2026-07-01 18:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:23.784595417 +0000 UTC m=+122.019549684" watchObservedRunningTime="2026-07-01 18:39:23.786965702 +0000 UTC m=+122.021919976"
Jul 01 18:39:23.789967 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.789946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6d2\" (UniqueName: \"kubernetes.io/projected/cc539bb8-2af5-4c86-83fc-d5eb5046950a-kube-api-access-kb6d2\") pod \"thanos-querier-645f96b865-npct9\" (UID: \"cc539bb8-2af5-4c86-83fc-d5eb5046950a\") " pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:23.797694 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.797675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-645f96b865-npct9"]
Jul 01 18:39:23.810964 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.810938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:23.967069 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:23.966975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:24.124736 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.124698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:24.146109 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.146069    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-645f96b865-npct9"]
Jul 01 18:39:24.151949 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:24.151919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc539bb8_2af5_4c86_83fc_d5eb5046950a.slice/crio-d59caf27cdcad773b63ca14ee4b6c26de2dcca5c13d500ff67fcb53220f14cb1 WatchSource:0}: Error finding container d59caf27cdcad773b63ca14ee4b6c26de2dcca5c13d500ff67fcb53220f14cb1: Status 404 returned error can't find the container with id d59caf27cdcad773b63ca14ee4b6c26de2dcca5c13d500ff67fcb53220f14cb1
Jul 01 18:39:24.248439 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.248402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:24.443824 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.443777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:24.631753 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.631651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:24.721392 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.721360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"d59caf27cdcad773b63ca14ee4b6c26de2dcca5c13d500ff67fcb53220f14cb1"}
Jul 01 18:39:24.748997 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.748965    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"]
Jul 01 18:39:24.752648 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.752626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:24.756794 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.756770    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n8nhs\""
Jul 01 18:39:24.757499 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.757476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 01 18:39:24.782511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.782483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hn9b\" (UniqueName: \"kubernetes.io/projected/ab61fa60-f027-44bf-a598-9ae1532455b3-kube-api-access-9hn9b\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:24.782648 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.782572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab61fa60-f027-44bf-a598-9ae1532455b3-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:24.783050 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.783029    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"]
Jul 01 18:39:24.799260 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.799232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:24.884645 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.883701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hn9b\" (UniqueName: \"kubernetes.io/projected/ab61fa60-f027-44bf-a598-9ae1532455b3-kube-api-access-9hn9b\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:24.884645 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.883827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab61fa60-f027-44bf-a598-9ae1532455b3-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:24.884645 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:39:24.883997    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 01 18:39:24.884645 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:39:24.884057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab61fa60-f027-44bf-a598-9ae1532455b3-monitoring-plugin-cert podName:ab61fa60-f027-44bf-a598-9ae1532455b3 nodeName:}" failed. No retries permitted until 2026-07-01 18:39:25.384038503 +0000 UTC m=+123.618992770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ab61fa60-f027-44bf-a598-9ae1532455b3-monitoring-plugin-cert") pod "monitoring-plugin-cd8656b8-qbvsz" (UID: "ab61fa60-f027-44bf-a598-9ae1532455b3") : secret "monitoring-plugin-cert" not found
Jul 01 18:39:24.923244 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.923205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hn9b\" (UniqueName: \"kubernetes.io/projected/ab61fa60-f027-44bf-a598-9ae1532455b3-kube-api-access-9hn9b\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:24.973322 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:24.973290    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:25.191467 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:25.191386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:25.349531 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:25.349499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:25.389259 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:25.388231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab61fa60-f027-44bf-a598-9ae1532455b3-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:25.391787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:25.391762    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab61fa60-f027-44bf-a598-9ae1532455b3-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-qbvsz\" (UID: \"ab61fa60-f027-44bf-a598-9ae1532455b3\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:25.664860 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:25.664794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:26.463850 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.463817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"]
Jul 01 18:39:26.464240 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:26.464076    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab61fa60_f027_44bf_a598_9ae1532455b3.slice/crio-e8865a695681ea0e298e07116672fd238014b9a7927733da17fef8e435f1946c WatchSource:0}: Error finding container e8865a695681ea0e298e07116672fd238014b9a7927733da17fef8e435f1946c: Status 404 returned error can't find the container with id e8865a695681ea0e298e07116672fd238014b9a7927733da17fef8e435f1946c
Jul 01 18:39:26.733700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.733664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"40ce8f9966d365dc0e79795b1ab2ef9b8adb81ec96afd9263c08a4c86f307032"}
Jul 01 18:39:26.733925 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.733790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"05e0b46b8975b465942ee96456f139eb392e5f0f08b04f9a902294d73cf33af3"}
Jul 01 18:39:26.733925 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.733858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"6d1a74fc17e85f9df00043b9975b3fba31fc6911fd5df693ef4860256295372d"}
Jul 01 18:39:26.735080 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.735055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz" event={"ID":"ab61fa60-f027-44bf-a598-9ae1532455b3","Type":"ContainerStarted","Data":"e8865a695681ea0e298e07116672fd238014b9a7927733da17fef8e435f1946c"}
Jul 01 18:39:26.770617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.770581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:26.916231 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:26.916193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:27.743152 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.743070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"e7d86cef721e3616956eb39438ce1d60df1a8f3c2c205cacf090ad939d98d0e0"}
Jul 01 18:39:27.743152 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.743157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"00fc0de84f76f028052a0bd30eda15b1af9e6e481f63aa1a1e330a2eb5e08cdd"}
Jul 01 18:39:27.743669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.743197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" event={"ID":"cc539bb8-2af5-4c86-83fc-d5eb5046950a","Type":"ContainerStarted","Data":"48c7b7c9adc4d6fd1b38dbf64b7aa7b0d8ec67577abea37546d8a798d3436c22"}
Jul 01 18:39:27.743669 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.743653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:27.801502 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.801447    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-645f96b865-npct9" podStartSLOduration=2.107273938 podStartE2EDuration="4.801430946s" totalImagesPullingTime="2.694157008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:27.799958626 +0000 UTC m=+126.034912911" watchObservedRunningTime="2026-07-01 18:39:27.801430946 +0000 UTC m=+126.036385231"
Jul 01 18:39:27.927109 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.927073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:27.928034 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.928008    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:39:27.932960 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.932939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:27.942403 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942286    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 18:39:27.942403 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 18:39:27.942563 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942539    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 18:39:27.942616 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942600    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 18:39:27.942997 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 18:39:27.942997 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 18:39:27.942997 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.942955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 18:39:27.943202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.943097    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 18:39:27.943255 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.943206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 18:39:27.943307 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.943287    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 18:39:27.943489 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.943471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 18:39:27.943564 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.943468    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 18:39:27.949297 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.949273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-16iv0utfoadb9\""
Jul 01 18:39:27.949394 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.949331    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mp5c\""
Jul 01 18:39:27.958202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.957996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 18:39:27.965619 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.965579    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 18:39:27.976353 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:27.976333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 18:39:28.003150 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.003087    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:39:28.014788 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.014764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.014919 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.014795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.014919 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.014840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.014919 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.014866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.014919 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.014888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015140 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.014997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015140 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015140 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015280 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015280 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015280 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28km\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-kube-api-access-w28km\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015424 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config-out\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015424 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015496 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015496 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015496 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-web-config\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.015756 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.015634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116204 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116204 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-web-config\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28km\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-kube-api-access-w28km\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config-out\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.116787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.116744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.118328 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.117343    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.118328 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.117421    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.118328 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.117438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.118328 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.118025    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.118328 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.118051    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.120681 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.119926    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.121787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.121462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.123596 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.123256    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config-out\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.123596 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.123339    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.123596 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.123404    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.123842 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.123726    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.123842 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.123781    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.123988 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.123860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.124192 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.124171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.124364 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.124311    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-web-config\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.124364 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.124226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.124900 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.124860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.125073 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.125013    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.126069 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.126037    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.136304 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.136254    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28km\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-kube-api-access-w28km\") pod \"prometheus-k8s-0\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.245548 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.245518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:28.455946 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.455913    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:39:28.459370 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:28.459346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23285cdf_d1ba_4a2b_8dc3_a5ef10e7d839.slice/crio-aaffae5c79d2708fc01412d4a5be68656bdfda09e6b57f9a1cdc3802ce908b6b WatchSource:0}: Error finding container aaffae5c79d2708fc01412d4a5be68656bdfda09e6b57f9a1cdc3802ce908b6b: Status 404 returned error can't find the container with id aaffae5c79d2708fc01412d4a5be68656bdfda09e6b57f9a1cdc3802ce908b6b
Jul 01 18:39:28.517631 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.517569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:28.619909 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.619874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:28.749567 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.749529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"aaffae5c79d2708fc01412d4a5be68656bdfda09e6b57f9a1cdc3802ce908b6b"}
Jul 01 18:39:28.751358 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.751326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz" event={"ID":"ab61fa60-f027-44bf-a598-9ae1532455b3","Type":"ContainerStarted","Data":"622cb124c0201433442345a6b44327a773a3b325868837536e16da4f15d4c7a8"}
Jul 01 18:39:28.751728 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.751709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:28.757370 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.757354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz"
Jul 01 18:39:28.785103 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:28.785016    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-qbvsz" podStartSLOduration=2.933286044 podStartE2EDuration="4.785003351s" totalImagesPullingTime="1.851717307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:28.783188651 +0000 UTC m=+127.018142918" watchObservedRunningTime="2026-07-01 18:39:28.785003351 +0000 UTC m=+127.019957634"
Jul 01 18:39:29.757429 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:29.757397    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652" exitCode=0
Jul 01 18:39:29.757849 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:29.757486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"}
Jul 01 18:39:32.057988 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.057942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:39:32.060665 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.060642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f6cb189-4786-4e41-bd91-0888314a0b65-metrics-certs\") pod \"network-metrics-daemon-xkqt7\" (UID: \"9f6cb189-4786-4e41-bd91-0888314a0b65\") " pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:39:32.234798 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.234570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plpqm\""
Jul 01 18:39:32.237225 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.236887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xkqt7"
Jul 01 18:39:32.419598 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.419501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:32.668506 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.668472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:32.866414 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:32.865935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:33.040088 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:33.040054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:33.690175 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:33.690143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xkqt7"]
Jul 01 18:39:33.759641 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:33.759613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-645f96b865-npct9"
Jul 01 18:39:33.852952 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:39:33.852850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6cb189_4786_4e41_bd91_0888314a0b65.slice/crio-1dd2b6f2e89b041e30e35532284bba8ea9395ac869e7cb785c5f1ca2fdefe14e WatchSource:0}: Error finding container 1dd2b6f2e89b041e30e35532284bba8ea9395ac869e7cb785c5f1ca2fdefe14e: Status 404 returned error can't find the container with id 1dd2b6f2e89b041e30e35532284bba8ea9395ac869e7cb785c5f1ca2fdefe14e
Jul 01 18:39:34.785114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.785077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"}
Jul 01 18:39:34.785595 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.785156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"}
Jul 01 18:39:34.785595 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.785199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"}
Jul 01 18:39:34.785595 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.785238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"}
Jul 01 18:39:34.785595 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.785262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"}
Jul 01 18:39:34.785595 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.785303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerStarted","Data":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"}
Jul 01 18:39:34.786438 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.786409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkqt7" event={"ID":"9f6cb189-4786-4e41-bd91-0888314a0b65","Type":"ContainerStarted","Data":"1dd2b6f2e89b041e30e35532284bba8ea9395ac869e7cb785c5f1ca2fdefe14e"}
Jul 01 18:39:34.856621 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:34.856545    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7386413750000003 podStartE2EDuration="7.856521267s" totalImagesPullingTime="5.097192651s" totalInitContainerRuntime="20.687241ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:34.852117823 +0000 UTC m=+133.087072149" watchObservedRunningTime="2026-07-01 18:39:34.856521267 +0000 UTC m=+133.091475534"
Jul 01 18:39:35.270114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:35.270077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:35.444704 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:35.444618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:35.793015 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:35.792910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkqt7" event={"ID":"9f6cb189-4786-4e41-bd91-0888314a0b65","Type":"ContainerStarted","Data":"8ab8daf602397d4d0b868d0e835fa2d716f349fe269cfdce287efe0404e51847"}
Jul 01 18:39:35.793015 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:35.793007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xkqt7" event={"ID":"9f6cb189-4786-4e41-bd91-0888314a0b65","Type":"ContainerStarted","Data":"035ef987c273364a95bdf06df3538e63f2c99f08994414860cac841b4ba5035a"}
Jul 01 18:39:35.825214 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:35.825150    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xkqt7" podStartSLOduration=132.662832409 podStartE2EDuration="2m13.825128794s" totalImagesPullingTime="1.162296385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:35.824030296 +0000 UTC m=+134.058984586" watchObservedRunningTime="2026-07-01 18:39:35.825128794 +0000 UTC m=+134.060083063"
Jul 01 18:39:36.029269 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:36.029230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:36.316880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:36.316849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:37.306568 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:37.306536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:37.432991 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:37.432954    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7949f867cb-75prg" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerName="registry" containerID="cri-o://3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79" gracePeriod=30
Jul 01 18:39:37.436502 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:37.436474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:37.639126 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:37.639050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:37.776797 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:37.776765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:38.245899 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:38.245865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:39:40.007538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:40.007503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:40.158108 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:40.158070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:40.278304 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:40.278220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:42.414413 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:42.414377    2577 patch_prober.go:28] interesting pod/image-registry-7949f867cb-75prg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.134.0.14:5000/healthz\": dial tcp 10.134.0.14:5000: connect: connection refused" start-of-body=
Jul 01 18:39:42.414828 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:42.414442    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7949f867cb-75prg" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.14:5000/healthz\": dial tcp 10.134.0.14:5000: connect: connection refused"
Jul 01 18:39:43.540903 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.540874    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:43.673792 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.673715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-certificates\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.673792 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.673751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-installation-pull-secrets\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.673792 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.673780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjd6h\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-kube-api-access-xjd6h\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.674063 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.673819    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-ca-trust-extracted\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.674162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.674136    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:43.675057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.674282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.675057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.674337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-image-registry-private-configuration\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.675057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.674384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-trusted-ca\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.675057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.674455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-bound-sa-token\") pod \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\" (UID: \"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d\") "
Jul 01 18:39:43.675057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.674743    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-certificates\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.675782 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.675751    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:39:43.677852 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.677676    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:43.677852 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.677704    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-kube-api-access-xjd6h" (OuterVolumeSpecName: "kube-api-access-xjd6h") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "kube-api-access-xjd6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:43.677852 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.677794    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:43.678199 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.678160    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:39:43.678514 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.678474    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:39:43.686142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.686116    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" (UID: "f2fe941b-22a0-4cea-90c9-d33f2fe11c7d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:39:43.775537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775505    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-bound-sa-token\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.775537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775533    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-installation-pull-secrets\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.775537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775544    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjd6h\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-kube-api-access-xjd6h\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.775782 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775554    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-ca-trust-extracted\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.775782 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775566    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-registry-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.775782 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775579    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-image-registry-private-configuration\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.775782 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.775587    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d-trusted-ca\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:39:43.833297 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.833264    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerID="3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79" exitCode=0
Jul 01 18:39:43.833480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.833333    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7949f867cb-75prg"
Jul 01 18:39:43.833480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.833355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7949f867cb-75prg" event={"ID":"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d","Type":"ContainerDied","Data":"3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79"}
Jul 01 18:39:43.833480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.833441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7949f867cb-75prg" event={"ID":"f2fe941b-22a0-4cea-90c9-d33f2fe11c7d","Type":"ContainerDied","Data":"535469dffd27979dfc8d71487e6be303899a0d15c773c30f775da61fb6314835"}
Jul 01 18:39:43.833480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.833468    2577 scope.go:117] "RemoveContainer" containerID="3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79"
Jul 01 18:39:43.835518 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.835479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-qkt6x" event={"ID":"f2fe2315-f9aa-426d-9adf-93ff2f0bd73f","Type":"ContainerStarted","Data":"3130098ae5acf2cc37ebd3d503ef884663bcd2bd689b9401bdbcc4afbcdaf944"}
Jul 01 18:39:43.835952 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.835934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:43.843845 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.843800    2577 scope.go:117] "RemoveContainer" containerID="3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79"
Jul 01 18:39:43.844097 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:39:43.844074    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79\": container with ID starting with 3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79 not found: ID does not exist" containerID="3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79"
Jul 01 18:39:43.844181 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.844103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79"} err="failed to get container status \"3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79\": rpc error: code = NotFound desc = could not find container \"3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79\": container with ID starting with 3a459c46fc2abd1bc07b8b7f710402ba8e4b21b9f1883e94d88b2a68d4bffc79 not found: ID does not exist"
Jul 01 18:39:43.847788 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.847769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-qkt6x"
Jul 01 18:39:43.916231 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.916197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7949f867cb-75prg"]
Jul 01 18:39:43.939545 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:43.939461    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7949f867cb-75prg"]
Jul 01 18:39:44.000716 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:44.000667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-qkt6x" podStartSLOduration=1.980964638 podStartE2EDuration="22.000649804s" totalImagesPullingTime="20.019685166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:39:43.998545862 +0000 UTC m=+142.233500152" watchObservedRunningTime="2026-07-01 18:39:44.000649804 +0000 UTC m=+142.235604070"
Jul 01 18:39:44.210456 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:44.210376    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" path="/var/lib/kubelet/pods/f2fe941b-22a0-4cea-90c9-d33f2fe11c7d/volumes"
Jul 01 18:39:45.037471 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:45.037431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:45.128052 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:45.128013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:49.705730 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:49.705694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:49.803494 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:49.803458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:49.888138 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:49.888107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:50.874735 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:50.874695    2577 generic.go:356] "Generic (PLEG): container finished" podID="4ca4e42f-afd3-466e-9b65-8958c18336a2" containerID="f9a0734a8eb36c4652e33d25f2386c1ccf5256886ba6512d6a43a9dbc3f4a1ec" exitCode=0
Jul 01 18:39:50.875224 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:50.874776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps" event={"ID":"4ca4e42f-afd3-466e-9b65-8958c18336a2","Type":"ContainerDied","Data":"f9a0734a8eb36c4652e33d25f2386c1ccf5256886ba6512d6a43a9dbc3f4a1ec"}
Jul 01 18:39:50.875224 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:50.875170    2577 scope.go:117] "RemoveContainer" containerID="f9a0734a8eb36c4652e33d25f2386c1ccf5256886ba6512d6a43a9dbc3f4a1ec"
Jul 01 18:39:51.768242 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:51.768209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:51.879620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:51.879586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:51.881595 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:51.881563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-p8vps" event={"ID":"4ca4e42f-afd3-466e-9b65-8958c18336a2","Type":"ContainerStarted","Data":"6c6575b43b694c79ba81768edf1b692a3dde09e9bb0fba761c1a8b41e96ea7b1"}
Jul 01 18:39:52.262681 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:52.262642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:52.369660 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:52.369633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:39:53.764950 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:39:53.764920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:03.160839 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:03.160785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:03.278124 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:03.278092    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:07.245966 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:07.245932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:07.351825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:07.351779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:28.246365 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:28.246325    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:28.333071 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:28.333042    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:29.045500 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:29.045471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:41.008318 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:41.008283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:41.399480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:41.399394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:46.445617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.445581    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:46.446498 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.446466    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="prometheus" containerID="cri-o://f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823" gracePeriod=600
Jul 01 18:40:46.447293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.446699    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-web" containerID="cri-o://d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c" gracePeriod=600
Jul 01 18:40:46.447293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.446725    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy" containerID="cri-o://1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7" gracePeriod=600
Jul 01 18:40:46.447293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.446742    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="config-reloader" containerID="cri-o://fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261" gracePeriod=600
Jul 01 18:40:46.447293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.446730    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="thanos-sidecar" containerID="cri-o://787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6" gracePeriod=600
Jul 01 18:40:46.447293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.446866    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-thanos" containerID="cri-o://10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f" gracePeriod=600
Jul 01 18:40:46.713954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.713583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:46.811116 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811083    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811116 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811124    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-db\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-thanos-prometheus-http-client-file\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-kubelet-serving-ca-bundle\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-trusted-ca-bundle\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-1\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w28km\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-kube-api-access-w28km\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-web-config\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-kube-rbac-proxy\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-metrics-client-ca\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811421    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-0\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811446    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-tls-assets\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811472    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config-out\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-grpc-tls\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-tls\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-2\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-serving-certs-ca-bundle\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.811729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.812209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-metrics-client-certs\") pod \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\" (UID: \"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839\") "
Jul 01 18:40:46.812209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811573    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.812209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.812130    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.812209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811615    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.812209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.811890    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.812977 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.812572    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:40:46.812977 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.812605    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.812977 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.812941    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.812977 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.812958    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:40:46.815341 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.815211    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.815341 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.815311    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.815341 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.815239    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.815559 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.815380    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.815817 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.815771    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config" (OuterVolumeSpecName: "config") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.816068 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.816037    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.816161 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.816128    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:46.816226 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.816202    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.816423 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.816407    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-kube-api-access-w28km" (OuterVolumeSpecName: "kube-api-access-w28km") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "kube-api-access-w28km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:40:46.816775 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.816754    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.817017 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.816996    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config-out" (OuterVolumeSpecName: "config-out") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:40:46.827315 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.827290    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-web-config" (OuterVolumeSpecName: "web-config") pod "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" (UID: "23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:40:46.913253 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913219    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913253 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913245    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-db\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913253 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913259    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913271    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913284    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913297    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28km\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-kube-api-access-w28km\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913342    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-web-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913355    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-kube-rbac-proxy\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913367    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-metrics-client-ca\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913379    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913391    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-tls-assets\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913402    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config-out\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913413    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913423    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-grpc-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913434    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913446    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913459    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913473    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:46.913538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:46.913487    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839-secret-metrics-client-certs\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:40:47.104272 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104235    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f" exitCode=0
Jul 01 18:40:47.104272 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104268    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7" exitCode=0
Jul 01 18:40:47.104272 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104275    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c" exitCode=0
Jul 01 18:40:47.104272 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104280    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6" exitCode=0
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104285    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261" exitCode=0
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104290    2577 generic.go:356] "Generic (PLEG): container finished" podID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823" exitCode=0
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104315    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"}
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"}
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"}
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"}
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104490    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"}
Jul 01 18:40:47.104541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"}
Jul 01 18:40:47.104971 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.104553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839","Type":"ContainerDied","Data":"aaffae5c79d2708fc01412d4a5be68656bdfda09e6b57f9a1cdc3802ce908b6b"}
Jul 01 18:40:47.112505 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.112484    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.119702 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.119684    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.126600 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.126582    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.128869 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.128848    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:47.134102 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.134081    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.135094 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.134990    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:47.141113 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.141095    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.149025 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.149006    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.156118 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.156100    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.156479 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.156457    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.156551 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.156488    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"} err="failed to get container status \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist"
Jul 01 18:40:47.156551 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.156515    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.156827 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.156793    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.156899 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.156833    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"} err="failed to get container status \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist"
Jul 01 18:40:47.156899 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.156848    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.157129 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.157108    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.157227 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157143    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"} err="failed to get container status \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist"
Jul 01 18:40:47.157227 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157165    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.157428 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.157409    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.157475 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157432    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"} err="failed to get container status \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist"
Jul 01 18:40:47.157475 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157446    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.157686 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.157670    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.157730 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157690    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"} err="failed to get container status \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist"
Jul 01 18:40:47.157730 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157707    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.157963 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.157931    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.158031 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.157982    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"} err="failed to get container status \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist"
Jul 01 18:40:47.158031 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158004    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.158243 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:40:47.158227    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.158286 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158248    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"} err="failed to get container status \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist"
Jul 01 18:40:47.158286 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158265    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.158459 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158441    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"} err="failed to get container status \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist"
Jul 01 18:40:47.158519 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158460    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.158663 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158647    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"} err="failed to get container status \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist"
Jul 01 18:40:47.158701 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158662    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.158862 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158841    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"} err="failed to get container status \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist"
Jul 01 18:40:47.158940 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.158862    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.159056 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159038    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"} err="failed to get container status \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist"
Jul 01 18:40:47.159103 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159057    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.159275 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159256    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"} err="failed to get container status \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist"
Jul 01 18:40:47.159319 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159277    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.159475 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159459    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"} err="failed to get container status \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist"
Jul 01 18:40:47.159544 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159478    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.159708 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159691    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"} err="failed to get container status \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist"
Jul 01 18:40:47.159752 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159709    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.159901 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159883    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"} err="failed to get container status \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist"
Jul 01 18:40:47.159972 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.159902    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.160127 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160111    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"} err="failed to get container status \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist"
Jul 01 18:40:47.160173 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160128    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.160320 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160303    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"} err="failed to get container status \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist"
Jul 01 18:40:47.160380 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160321    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.160517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160498    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"} err="failed to get container status \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist"
Jul 01 18:40:47.160566 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160519    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.160705 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160685    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"} err="failed to get container status \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist"
Jul 01 18:40:47.160749 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160706    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.160924 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160907    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"} err="failed to get container status \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist"
Jul 01 18:40:47.160975 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.160924    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.161114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161099    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"} err="failed to get container status \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist"
Jul 01 18:40:47.161114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161114    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.161312 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161292    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"} err="failed to get container status \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist"
Jul 01 18:40:47.161385 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161314    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.161523 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161503    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"} err="failed to get container status \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist"
Jul 01 18:40:47.161590 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161523    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.161704 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161684    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"} err="failed to get container status \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist"
Jul 01 18:40:47.161756 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161705    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.162000 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.161983    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"} err="failed to get container status \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist"
Jul 01 18:40:47.162054 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162000    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.162210 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162192    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"} err="failed to get container status \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist"
Jul 01 18:40:47.162255 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162211    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.162434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162419    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"} err="failed to get container status \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist"
Jul 01 18:40:47.162488 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162434    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.162645 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162628    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"} err="failed to get container status \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist"
Jul 01 18:40:47.162686 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162648    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.162882 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162863    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"} err="failed to get container status \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist"
Jul 01 18:40:47.162882 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.162881    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.163143 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163123    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"} err="failed to get container status \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist"
Jul 01 18:40:47.163143 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163140    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.163348 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163328    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"} err="failed to get container status \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist"
Jul 01 18:40:47.163399 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163347    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.163553 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163534    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"} err="failed to get container status \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist"
Jul 01 18:40:47.163631 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163555    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.163824 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"} err="failed to get container status \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist"
Jul 01 18:40:47.163917 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.163850    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.164207 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.164184    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"} err="failed to get container status \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist"
Jul 01 18:40:47.164207 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.164206    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.164461 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.164440    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"} err="failed to get container status \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist"
Jul 01 18:40:47.164517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.164463    2577 scope.go:117] "RemoveContainer" containerID="10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"
Jul 01 18:40:47.164706 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.164680    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f"} err="failed to get container status \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": rpc error: code = NotFound desc = could not find container \"10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f\": container with ID starting with 10784cc0fd03454aeb71be254f57c4bb96bdabb9fe4ed93fa6ef7c4ac73c8b6f not found: ID does not exist"
Jul 01 18:40:47.164706 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.164701    2577 scope.go:117] "RemoveContainer" containerID="1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"
Jul 01 18:40:47.165383 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.165359    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7"} err="failed to get container status \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": rpc error: code = NotFound desc = could not find container \"1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7\": container with ID starting with 1d8ee073504529fb42fd13acf8cf606dc7017ce63ea477a2c1a338e0715e47a7 not found: ID does not exist"
Jul 01 18:40:47.165494 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.165383    2577 scope.go:117] "RemoveContainer" containerID="d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"
Jul 01 18:40:47.165983 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.165945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c"} err="failed to get container status \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": rpc error: code = NotFound desc = could not find container \"d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c\": container with ID starting with d6c8c668ba2c8a485ea25d3edf161a9dd9a3916991e6bf6f01036d936b020e9c not found: ID does not exist"
Jul 01 18:40:47.165983 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.165974    2577 scope.go:117] "RemoveContainer" containerID="787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"
Jul 01 18:40:47.166251 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166231    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6"} err="failed to get container status \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": rpc error: code = NotFound desc = could not find container \"787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6\": container with ID starting with 787480670ec84438088f8c6fe3ea8006d37de5c499d987e6e8db61e17a3c6ff6 not found: ID does not exist"
Jul 01 18:40:47.166251 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166251    2577 scope.go:117] "RemoveContainer" containerID="fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"
Jul 01 18:40:47.166508 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166487    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261"} err="failed to get container status \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": rpc error: code = NotFound desc = could not find container \"fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261\": container with ID starting with fe7b76c128739551ddbbd47cb3cbb70e22ec115f4b368e6f741dcc701b950261 not found: ID does not exist"
Jul 01 18:40:47.166564 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166509    2577 scope.go:117] "RemoveContainer" containerID="f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"
Jul 01 18:40:47.166719 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166696    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823"} err="failed to get container status \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": rpc error: code = NotFound desc = could not find container \"f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823\": container with ID starting with f1cabc1c87f36c0356fb848cdd0f3b53ed300ad76779fd1430702e044599f823 not found: ID does not exist"
Jul 01 18:40:47.166719 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166718    2577 scope.go:117] "RemoveContainer" containerID="be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"
Jul 01 18:40:47.166856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166720    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:47.166941 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.166922    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652"} err="failed to get container status \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": rpc error: code = NotFound desc = could not find container \"be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652\": container with ID starting with be95d5154b01ea3e19024cd600c8e4471c82ff834f5887e5b6df92c76058c652 not found: ID does not exist"
Jul 01 18:40:47.167131 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167116    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-web"
Jul 01 18:40:47.167193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167135    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-web"
Jul 01 18:40:47.167193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167155    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerName="registry"
Jul 01 18:40:47.167193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167166    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerName="registry"
Jul 01 18:40:47.167193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167181    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="prometheus"
Jul 01 18:40:47.167193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167190    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="prometheus"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167201    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-thanos"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-thanos"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167223    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="thanos-sidecar"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="thanos-sidecar"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167249    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="init-config-reloader"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167257    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="init-config-reloader"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167271    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167280    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167290    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="config-reloader"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167298    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="config-reloader"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167377    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167391    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-web"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167400    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fe941b-22a0-4cea-90c9-d33f2fe11c7d" containerName="registry"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167411    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="kube-rbac-proxy-thanos"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167422    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="prometheus"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167431    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="config-reloader"
Jul 01 18:40:47.167434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.167438    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" containerName="thanos-sidecar"
Jul 01 18:40:47.171491 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.171472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.176058 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 01 18:40:47.176189 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176108    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 01 18:40:47.176364 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176349    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 01 18:40:47.176507 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176492    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 01 18:40:47.176550 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176521    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 01 18:40:47.176881 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176863    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 01 18:40:47.176999 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.176897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-16iv0utfoadb9\""
Jul 01 18:40:47.177047 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 01 18:40:47.177216 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177195    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 01 18:40:47.177370 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177228    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 01 18:40:47.177370 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177197    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 18:40:47.177370 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 01 18:40:47.177636 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177615    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mp5c\""
Jul 01 18:40:47.177976 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177956    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 01 18:40:47.178057 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.177956    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 01 18:40:47.180875 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.180796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 01 18:40:47.185225 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.185207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 01 18:40:47.190851 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.190832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:47.317200 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317200 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33e70a4a-bf63-4524-9040-78c6b60b2d1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8wz\" (UniqueName: \"kubernetes.io/projected/33e70a4a-bf63-4524-9040-78c6b60b2d1a-kube-api-access-nb8wz\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-config\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317407 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33e70a4a-bf63-4524-9040-78c6b60b2d1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.317617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.317613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.418699 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-config\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.418699 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.418699 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.418699 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33e70a4a-bf63-4524-9040-78c6b60b2d1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419026 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.418995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33e70a4a-bf63-4524-9040-78c6b60b2d1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419415 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8wz\" (UniqueName: \"kubernetes.io/projected/33e70a4a-bf63-4524-9040-78c6b60b2d1a-kube-api-access-nb8wz\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419871 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.419871 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.419600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.420263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.420239    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.421977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422021    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33e70a4a-bf63-4524-9040-78c6b60b2d1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422032    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422482 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422303    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422836 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422796    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.422987 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.422965    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-config\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.423067 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.423024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.423067 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.423044    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.423571 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.423548    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.424047 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.424024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33e70a4a-bf63-4524-9040-78c6b60b2d1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.424240 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.424221    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.424879 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.424863    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33e70a4a-bf63-4524-9040-78c6b60b2d1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.424928 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.424919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33e70a4a-bf63-4524-9040-78c6b60b2d1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.429163 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.429145    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8wz\" (UniqueName: \"kubernetes.io/projected/33e70a4a-bf63-4524-9040-78c6b60b2d1a-kube-api-access-nb8wz\") pod \"prometheus-k8s-0\" (UID: \"33e70a4a-bf63-4524-9040-78c6b60b2d1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.499109 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.498305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:47.647074 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:47.647044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 01 18:40:47.650381 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:40:47.650356    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e70a4a_bf63_4524_9040_78c6b60b2d1a.slice/crio-fe6082a67e24f17f289daebe24e82fb8d0fcbca2caaa830551f52963e900d107 WatchSource:0}: Error finding container fe6082a67e24f17f289daebe24e82fb8d0fcbca2caaa830551f52963e900d107: Status 404 returned error can't find the container with id fe6082a67e24f17f289daebe24e82fb8d0fcbca2caaa830551f52963e900d107
Jul 01 18:40:48.110722 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:48.110689    2577 generic.go:356] "Generic (PLEG): container finished" podID="33e70a4a-bf63-4524-9040-78c6b60b2d1a" containerID="d33e333aaba4720a6f75906cc017b9c0fc6e7e3c0b8022ead192a384f92b9ea5" exitCode=0
Jul 01 18:40:48.110907 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:48.110723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerDied","Data":"d33e333aaba4720a6f75906cc017b9c0fc6e7e3c0b8022ead192a384f92b9ea5"}
Jul 01 18:40:48.110907 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:48.110786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"fe6082a67e24f17f289daebe24e82fb8d0fcbca2caaa830551f52963e900d107"}
Jul 01 18:40:48.212788 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:48.212751    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839" path="/var/lib/kubelet/pods/23285cdf-d1ba-4a2b-8dc3-a5ef10e7d839/volumes"
Jul 01 18:40:49.118131 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.118100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"5e78c5a87305a4add0d3088c17cbc97be3f0e14fcb3724c6057acdd9d4b29111"}
Jul 01 18:40:49.118517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.118164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"a07c942b38d646f0d81cd7f54472671fd8c80cc259f9cf94227fdb84ced37f6b"}
Jul 01 18:40:49.118517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.118191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"45e73b7940c8800ed0726fb8c602209f32373a05684d6f878cd9088e6e9813a6"}
Jul 01 18:40:49.118517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.118215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"03070c6b719cdf6e818afe32ae0f3c9369371d25bdfc1453bda192afcfb489e4"}
Jul 01 18:40:49.118517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.118240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"d09cc264ea1a23e03e12f9e0505c17318fb2c4556ad5c9088421f66f82075933"}
Jul 01 18:40:49.118517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.118265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33e70a4a-bf63-4524-9040-78c6b60b2d1a","Type":"ContainerStarted","Data":"a743f47b77b9cbb30138b6c6d309bd347017fa9aff7ab6579d376e289c2a5354"}
Jul 01 18:40:49.155331 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:49.155273    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.135295496 podStartE2EDuration="2.155255337s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.959841ms" isStatefulPod=false podCreationTimestamp="2026-07-01 18:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:40:49.15347075 +0000 UTC m=+207.388425016" watchObservedRunningTime="2026-07-01 18:40:49.155255337 +0000 UTC m=+207.390209603"
Jul 01 18:40:50.228129 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:50.228097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:50.368434 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:50.368395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:50.540011 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:50.539929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:50.971562 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:50.971533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:51.043703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:51.043671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:40:52.499409 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:52.499370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:40:53.592864 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:40:53.592822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:01.496450 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:01.496405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:01.603403 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:01.603363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:01.707219 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:01.707183    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:17.462231 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:17.462177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:37.690093 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:37.690060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:37.757726 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:37.757695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:47.499188 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:47.499090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:41:47.514333 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:47.514304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:41:48.375984 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:48.375959    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 01 18:41:56.286737 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:56.286702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:41:56.366136 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:41:56.366105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:08.379447 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:08.379413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:08.480142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:08.480114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:22.144005 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.143979    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:42:22.146781 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.146756    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:42:22.150368 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.150340    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:42:22.153660 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.153642    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:42:22.156467 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.156450    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 18:42:22.271742 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.271708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:22.366617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:22.366581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:23.116734 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:23.116697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:23.221634 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:23.221600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:33.874247 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:33.874198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:33.979357 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:33.979300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:34.074773 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:34.074728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:50.402288 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:50.402258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:42:51.850192 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:42:51.850134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:18.661267 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:18.661196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:18.756080 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:18.756047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:25.608310 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:25.608276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:25.906030 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:25.905938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:26.149224 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:26.149195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:26.248387 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:26.248360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:26.314517 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:26.314487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:26.805480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:26.805445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:26.890948 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:26.890910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:27.154922 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:27.154841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:27.341198 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:27.341164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:27.551406 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:27.551366    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:27.744987 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:27.744942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:43:27.859481 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:43:27.859402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:33.283868 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:33.283839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:44.962758 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:44.962673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:45.086987 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:45.086959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:45.235129 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:45.235100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:45.366198 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:45.366146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:45.910495 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:45.910457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:46.007215 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:46.007169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:44:46.112694 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:44:46.112659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:15.135640 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:15.135608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:57.412537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:57.412506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:57.517202 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:57.517176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:57.627618 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:57.627586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:57.740091 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:57.740059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:58.388885 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:58.388850    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:45:58.488360 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:45:58.488328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:46:14.496340 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.496263    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4n865"]
Jul 01 18:46:14.498516 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.498501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4n865"
Jul 01 18:46:14.501463 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.501438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 18:46:14.501596 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.501471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 18:46:14.501690 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.501678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 18:46:14.502665 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.502650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-78g7k\""
Jul 01 18:46:14.514259 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.514233    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4n865"]
Jul 01 18:46:14.631797 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.631769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8p67\" (UniqueName: \"kubernetes.io/projected/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-kube-api-access-g8p67\") pod \"s3-init-4n865\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") " pod="kserve/s3-init-4n865"
Jul 01 18:46:14.631797 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.631798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-model-cache\") pod \"s3-init-4n865\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") " pod="kserve/s3-init-4n865"
Jul 01 18:46:14.733281 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.733237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8p67\" (UniqueName: \"kubernetes.io/projected/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-kube-api-access-g8p67\") pod \"s3-init-4n865\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") " pod="kserve/s3-init-4n865"
Jul 01 18:46:14.733281 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.733281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-model-cache\") pod \"s3-init-4n865\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") " pod="kserve/s3-init-4n865"
Jul 01 18:46:14.733619 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.733603    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-model-cache\") pod \"s3-init-4n865\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") " pod="kserve/s3-init-4n865"
Jul 01 18:46:14.743014 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.742982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8p67\" (UniqueName: \"kubernetes.io/projected/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-kube-api-access-g8p67\") pod \"s3-init-4n865\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") " pod="kserve/s3-init-4n865"
Jul 01 18:46:14.815966 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.815877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4n865"
Jul 01 18:46:14.948902 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.948872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4n865"]
Jul 01 18:46:14.952292 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:46:14.952262    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e819c7b_bfb6_4f40_9a29_3d06239a26ea.slice/crio-3454c1e1ec1f8bfd568363d386167ef11b8805d2754bd5b355e433c61caf7320 WatchSource:0}: Error finding container 3454c1e1ec1f8bfd568363d386167ef11b8805d2754bd5b355e433c61caf7320: Status 404 returned error can't find the container with id 3454c1e1ec1f8bfd568363d386167ef11b8805d2754bd5b355e433c61caf7320
Jul 01 18:46:14.954178 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:14.954166    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:46:15.429015 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:15.428981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4n865" event={"ID":"0e819c7b-bfb6-4f40-9a29-3d06239a26ea","Type":"ContainerStarted","Data":"3454c1e1ec1f8bfd568363d386167ef11b8805d2754bd5b355e433c61caf7320"}
Jul 01 18:46:19.447532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:19.447493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4n865" event={"ID":"0e819c7b-bfb6-4f40-9a29-3d06239a26ea","Type":"ContainerStarted","Data":"b6f95599d7164521ca30defd3d33e6a1541b99c54cabdc761a79670287a09c54"}
Jul 01 18:46:21.457713 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:21.457684    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerID="b6f95599d7164521ca30defd3d33e6a1541b99c54cabdc761a79670287a09c54" exitCode=0
Jul 01 18:46:21.458142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:21.457767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4n865" event={"ID":"0e819c7b-bfb6-4f40-9a29-3d06239a26ea","Type":"ContainerDied","Data":"b6f95599d7164521ca30defd3d33e6a1541b99c54cabdc761a79670287a09c54"}
Jul 01 18:46:25.971048 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:25.971024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 18:46:26.482879 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:26.482846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4n865" event={"ID":"0e819c7b-bfb6-4f40-9a29-3d06239a26ea","Type":"ContainerStarted","Data":"481b7def68a8781dc7b1e2e9ba13af4cf3b96267a804c659bda59f00da3b894d"}
Jul 01 18:46:26.508706 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:46:26.508649    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-4n865" podStartSLOduration=2.199701218 podStartE2EDuration="12.508633393s" totalImagesPullingTime="8.831654454s" totalInitContainerRuntime="1.477277721s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:46:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:46:26.507019443 +0000 UTC m=+544.741973709" watchObservedRunningTime="2026-07-01 18:46:26.508633393 +0000 UTC m=+544.743587659"
Jul 01 18:47:22.173313 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:22.173286    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:47:22.173825 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:22.173704    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:47:22.178445 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:22.178422    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:47:22.178676 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:22.178655    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:47:31.744736 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:31.744704    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerID="481b7def68a8781dc7b1e2e9ba13af4cf3b96267a804c659bda59f00da3b894d" exitCode=0
Jul 01 18:47:31.745158 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:31.744765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4n865" event={"ID":"0e819c7b-bfb6-4f40-9a29-3d06239a26ea","Type":"ContainerDied","Data":"481b7def68a8781dc7b1e2e9ba13af4cf3b96267a804c659bda59f00da3b894d"}
Jul 01 18:47:32.872747 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:32.872723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4n865"
Jul 01 18:47:32.973910 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:32.973870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8p67\" (UniqueName: \"kubernetes.io/projected/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-kube-api-access-g8p67\") pod \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") "
Jul 01 18:47:32.974072 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:32.973977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-model-cache\") pod \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\" (UID: \"0e819c7b-bfb6-4f40-9a29-3d06239a26ea\") "
Jul 01 18:47:32.976009 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:32.975974    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-kube-api-access-g8p67" (OuterVolumeSpecName: "kube-api-access-g8p67") pod "0e819c7b-bfb6-4f40-9a29-3d06239a26ea" (UID: "0e819c7b-bfb6-4f40-9a29-3d06239a26ea"). InnerVolumeSpecName "kube-api-access-g8p67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:47:32.992256 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:32.992225    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e819c7b-bfb6-4f40-9a29-3d06239a26ea" (UID: "0e819c7b-bfb6-4f40-9a29-3d06239a26ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:47:33.075562 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:33.075480    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-model-cache\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:47:33.075562 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:33.075515    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8p67\" (UniqueName: \"kubernetes.io/projected/0e819c7b-bfb6-4f40-9a29-3d06239a26ea-kube-api-access-g8p67\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:47:33.755065 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:33.755035    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4n865"
Jul 01 18:47:33.755065 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:33.755047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4n865" event={"ID":"0e819c7b-bfb6-4f40-9a29-3d06239a26ea","Type":"ContainerDied","Data":"3454c1e1ec1f8bfd568363d386167ef11b8805d2754bd5b355e433c61caf7320"}
Jul 01 18:47:33.755310 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:33.755083    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3454c1e1ec1f8bfd568363d386167ef11b8805d2754bd5b355e433c61caf7320"
Jul 01 18:47:34.584982 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:34.584950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:47:40.675070 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:40.675035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:47:40.749293 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:40.749257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:47:42.993220 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.993187    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"]
Jul 01 18:47:42.993578 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.993494    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerName="download-hf-model"
Jul 01 18:47:42.993578 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.993505    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerName="download-hf-model"
Jul 01 18:47:42.993578 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.993519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerName="s3-init"
Jul 01 18:47:42.993578 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.993525    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerName="s3-init"
Jul 01 18:47:42.993705 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.993581    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e819c7b-bfb6-4f40-9a29-3d06239a26ea" containerName="s3-init"
Jul 01 18:47:42.995768 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.995752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:42.998352 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.998332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 01 18:47:42.999248 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.999225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-78g7k\""
Jul 01 18:47:42.999391 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.999278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 18:47:42.999391 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.999323    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 01 18:47:42.999391 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:42.999284    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 18:47:43.004194 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.004174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"]
Jul 01 18:47:43.058824 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.058777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29wn\" (UniqueName: \"kubernetes.io/projected/41b69c0c-500c-4997-b3fb-46461ab0e6f2-kube-api-access-d29wn\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.059035 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.058857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/41b69c0c-500c-4997-b3fb-46461ab0e6f2-data\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.059035 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.058897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/41b69c0c-500c-4997-b3fb-46461ab0e6f2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.160209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.160176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29wn\" (UniqueName: \"kubernetes.io/projected/41b69c0c-500c-4997-b3fb-46461ab0e6f2-kube-api-access-d29wn\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.160369 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.160220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/41b69c0c-500c-4997-b3fb-46461ab0e6f2-data\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.160369 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.160244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/41b69c0c-500c-4997-b3fb-46461ab0e6f2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.160559 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.160539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/41b69c0c-500c-4997-b3fb-46461ab0e6f2-data\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.162569 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.162551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/41b69c0c-500c-4997-b3fb-46461ab0e6f2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.170062 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.170026    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29wn\" (UniqueName: \"kubernetes.io/projected/41b69c0c-500c-4997-b3fb-46461ab0e6f2-kube-api-access-d29wn\") pod \"seaweedfs-tls-custom-8b44d767d-qdtc5\" (UID: \"41b69c0c-500c-4997-b3fb-46461ab0e6f2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.305668 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.305584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"
Jul 01 18:47:43.443120 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.443087    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-qdtc5"]
Jul 01 18:47:43.447140 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:47:43.447115    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b69c0c_500c_4997_b3fb_46461ab0e6f2.slice/crio-37f3dfc6ab20dd14eb3f06aab5fea81ea47fd63374a20e39c6e93b10e42bde0a WatchSource:0}: Error finding container 37f3dfc6ab20dd14eb3f06aab5fea81ea47fd63374a20e39c6e93b10e42bde0a: Status 404 returned error can't find the container with id 37f3dfc6ab20dd14eb3f06aab5fea81ea47fd63374a20e39c6e93b10e42bde0a
Jul 01 18:47:43.796242 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:43.796200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5" event={"ID":"41b69c0c-500c-4997-b3fb-46461ab0e6f2","Type":"ContainerStarted","Data":"37f3dfc6ab20dd14eb3f06aab5fea81ea47fd63374a20e39c6e93b10e42bde0a"}
Jul 01 18:47:46.810013 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:46.809985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5" event={"ID":"41b69c0c-500c-4997-b3fb-46461ab0e6f2","Type":"ContainerStarted","Data":"2f7776d165f079a4d10f5f697022275242a2b37cfa1f69a52ff1546466acf9d8"}
Jul 01 18:47:46.828599 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:46.828550    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-qdtc5" podStartSLOduration=2.214192806 podStartE2EDuration="4.828534779s" totalImagesPullingTime="2.614341973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:47:46.826798923 +0000 UTC m=+625.061753200" watchObservedRunningTime="2026-07-01 18:47:46.828534779 +0000 UTC m=+625.063489044"
Jul 01 18:47:47.186886 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.186799    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-wtcxn"]
Jul 01 18:47:47.189794 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.189779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.206414 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.206391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-wtcxn"]
Jul 01 18:47:47.301348 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.301317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4rk\" (UniqueName: \"kubernetes.io/projected/4c2bed35-c921-41dd-a235-38a4d6703c66-kube-api-access-5d4rk\") pod \"s3-tls-init-custom-wtcxn\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") " pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.301521 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.301363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2bed35-c921-41dd-a235-38a4d6703c66-model-cache\") pod \"s3-tls-init-custom-wtcxn\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") " pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.402773 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.402733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4rk\" (UniqueName: \"kubernetes.io/projected/4c2bed35-c921-41dd-a235-38a4d6703c66-kube-api-access-5d4rk\") pod \"s3-tls-init-custom-wtcxn\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") " pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.402974 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.402785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2bed35-c921-41dd-a235-38a4d6703c66-model-cache\") pod \"s3-tls-init-custom-wtcxn\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") " pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.403209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.403189    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2bed35-c921-41dd-a235-38a4d6703c66-model-cache\") pod \"s3-tls-init-custom-wtcxn\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") " pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.413621 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.413588    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4rk\" (UniqueName: \"kubernetes.io/projected/4c2bed35-c921-41dd-a235-38a4d6703c66-kube-api-access-5d4rk\") pod \"s3-tls-init-custom-wtcxn\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") " pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.514343 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.514312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:47.640867 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.640841    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-wtcxn"]
Jul 01 18:47:47.643320 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:47:47.643295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2bed35_c921_41dd_a235_38a4d6703c66.slice/crio-f6caad8202c0a669b1f754f107badc780786419376b8d66863cf770fc8b4effc WatchSource:0}: Error finding container f6caad8202c0a669b1f754f107badc780786419376b8d66863cf770fc8b4effc: Status 404 returned error can't find the container with id f6caad8202c0a669b1f754f107badc780786419376b8d66863cf770fc8b4effc
Jul 01 18:47:47.815742 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.815646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wtcxn" event={"ID":"4c2bed35-c921-41dd-a235-38a4d6703c66","Type":"ContainerStarted","Data":"86dafbf85b3914f84bc378d07fbb4a2ed1fb605b080dfe2795c1555c89e5b34a"}
Jul 01 18:47:47.815742 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:47.815694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wtcxn" event={"ID":"4c2bed35-c921-41dd-a235-38a4d6703c66","Type":"ContainerStarted","Data":"f6caad8202c0a669b1f754f107badc780786419376b8d66863cf770fc8b4effc"}
Jul 01 18:47:49.470741 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:49.470702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:47:49.825818 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:49.825785    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerID="86dafbf85b3914f84bc378d07fbb4a2ed1fb605b080dfe2795c1555c89e5b34a" exitCode=0
Jul 01 18:47:49.825981 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:49.825858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wtcxn" event={"ID":"4c2bed35-c921-41dd-a235-38a4d6703c66","Type":"ContainerDied","Data":"86dafbf85b3914f84bc378d07fbb4a2ed1fb605b080dfe2795c1555c89e5b34a"}
Jul 01 18:47:50.836796 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:50.836761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wtcxn" event={"ID":"4c2bed35-c921-41dd-a235-38a4d6703c66","Type":"ContainerStarted","Data":"31b9fd2e397b466676afb73ab31cb884928c1e784aa61e870df35b887f093f24"}
Jul 01 18:47:50.856269 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:50.856199    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-wtcxn" podStartSLOduration=2.188011838 podStartE2EDuration="3.856181319s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.668169481s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:47:50.855420928 +0000 UTC m=+629.090375222" watchObservedRunningTime="2026-07-01 18:47:50.856181319 +0000 UTC m=+629.091135611"
Jul 01 18:47:57.866253 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:57.866220    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerID="31b9fd2e397b466676afb73ab31cb884928c1e784aa61e870df35b887f093f24" exitCode=0
Jul 01 18:47:57.866253 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:57.866262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wtcxn" event={"ID":"4c2bed35-c921-41dd-a235-38a4d6703c66","Type":"ContainerDied","Data":"31b9fd2e397b466676afb73ab31cb884928c1e784aa61e870df35b887f093f24"}
Jul 01 18:47:59.030559 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.030535    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:59.204274 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.204197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2bed35-c921-41dd-a235-38a4d6703c66-model-cache\") pod \"4c2bed35-c921-41dd-a235-38a4d6703c66\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") "
Jul 01 18:47:59.204448 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.204284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d4rk\" (UniqueName: \"kubernetes.io/projected/4c2bed35-c921-41dd-a235-38a4d6703c66-kube-api-access-5d4rk\") pod \"4c2bed35-c921-41dd-a235-38a4d6703c66\" (UID: \"4c2bed35-c921-41dd-a235-38a4d6703c66\") "
Jul 01 18:47:59.206345 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.206311    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2bed35-c921-41dd-a235-38a4d6703c66-kube-api-access-5d4rk" (OuterVolumeSpecName: "kube-api-access-5d4rk") pod "4c2bed35-c921-41dd-a235-38a4d6703c66" (UID: "4c2bed35-c921-41dd-a235-38a4d6703c66"). InnerVolumeSpecName "kube-api-access-5d4rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:47:59.226140 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.226111    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2bed35-c921-41dd-a235-38a4d6703c66-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c2bed35-c921-41dd-a235-38a4d6703c66" (UID: "4c2bed35-c921-41dd-a235-38a4d6703c66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:47:59.305346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.305319    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4rk\" (UniqueName: \"kubernetes.io/projected/4c2bed35-c921-41dd-a235-38a4d6703c66-kube-api-access-5d4rk\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:47:59.305346 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.305343    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2bed35-c921-41dd-a235-38a4d6703c66-model-cache\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:47:59.876283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.876254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-wtcxn"
Jul 01 18:47:59.876283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.876274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-wtcxn" event={"ID":"4c2bed35-c921-41dd-a235-38a4d6703c66","Type":"ContainerDied","Data":"f6caad8202c0a669b1f754f107badc780786419376b8d66863cf770fc8b4effc"}
Jul 01 18:47:59.876492 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:47:59.876312    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6caad8202c0a669b1f754f107badc780786419376b8d66863cf770fc8b4effc"
Jul 01 18:48:00.533278 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.533246    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"]
Jul 01 18:48:00.533624 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.533596    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerName="s3-tls-init-custom"
Jul 01 18:48:00.533624 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.533606    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerName="s3-tls-init-custom"
Jul 01 18:48:00.533624 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.533616    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerName="download-hf-model"
Jul 01 18:48:00.533624 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.533623    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerName="download-hf-model"
Jul 01 18:48:00.533753 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.533673    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2bed35-c921-41dd-a235-38a4d6703c66" containerName="s3-tls-init-custom"
Jul 01 18:48:00.535686 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.535671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.538495 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.538477    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 01 18:48:00.538730 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.538718    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 01 18:48:00.543432 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.543409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:00.545318 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.545300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"]
Jul 01 18:48:00.715793 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.715759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.715945 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.715830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.715945 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.715876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxn5g\" (UniqueName: \"kubernetes.io/projected/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-kube-api-access-rxn5g\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.817150 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.817067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxn5g\" (UniqueName: \"kubernetes.io/projected/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-kube-api-access-rxn5g\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.817284 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.817152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.817284 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.817187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.817557 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.817541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-data\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.819529 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.819513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.827184 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.827160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxn5g\" (UniqueName: \"kubernetes.io/projected/73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0-kube-api-access-rxn5g\") pod \"seaweedfs-tls-serving-67b7477c6d-hpwrt\" (UID: \"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.844981 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.844957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"
Jul 01 18:48:00.971912 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:00.971889    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt"]
Jul 01 18:48:00.974334 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:48:00.974308    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e0e2ff_145e_49b5_bf44_0a3c1f4ef8f0.slice/crio-cf8107f5038d14f8052ef9e965327846c8d6cc34b4680cb56ae710b969663b11 WatchSource:0}: Error finding container cf8107f5038d14f8052ef9e965327846c8d6cc34b4680cb56ae710b969663b11: Status 404 returned error can't find the container with id cf8107f5038d14f8052ef9e965327846c8d6cc34b4680cb56ae710b969663b11
Jul 01 18:48:01.885819 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:01.885784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt" event={"ID":"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0","Type":"ContainerStarted","Data":"4559436a9128e1c3d0fc60dd4135d8df9b3f24c8fe21eb7850cc75f53f6a25a1"}
Jul 01 18:48:01.886190 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:01.885847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt" event={"ID":"73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0","Type":"ContainerStarted","Data":"cf8107f5038d14f8052ef9e965327846c8d6cc34b4680cb56ae710b969663b11"}
Jul 01 18:48:01.903524 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:01.903480    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-hpwrt" podStartSLOduration=1.629666588 podStartE2EDuration="1.903465493s" totalImagesPullingTime="273.798905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:48:01.902942203 +0000 UTC m=+640.137896475" watchObservedRunningTime="2026-07-01 18:48:01.903465493 +0000 UTC m=+640.138419781"
Jul 01 18:48:02.311460 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.311430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:02.489051 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.489020    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-d66b8"]
Jul 01 18:48:02.492706 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.492683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.504545 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.504521    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-d66b8"]
Jul 01 18:48:02.633480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.633389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d47e275-d2e7-470a-8a65-de73969ee068-model-cache\") pod \"s3-tls-init-serving-d66b8\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") " pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.633480 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.633421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjs2\" (UniqueName: \"kubernetes.io/projected/3d47e275-d2e7-470a-8a65-de73969ee068-kube-api-access-pdjs2\") pod \"s3-tls-init-serving-d66b8\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") " pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.734219 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.734186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d47e275-d2e7-470a-8a65-de73969ee068-model-cache\") pod \"s3-tls-init-serving-d66b8\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") " pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.734219 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.734221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjs2\" (UniqueName: \"kubernetes.io/projected/3d47e275-d2e7-470a-8a65-de73969ee068-kube-api-access-pdjs2\") pod \"s3-tls-init-serving-d66b8\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") " pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.734559 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.734541    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d47e275-d2e7-470a-8a65-de73969ee068-model-cache\") pod \"s3-tls-init-serving-d66b8\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") " pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.743043 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.743012    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjs2\" (UniqueName: \"kubernetes.io/projected/3d47e275-d2e7-470a-8a65-de73969ee068-kube-api-access-pdjs2\") pod \"s3-tls-init-serving-d66b8\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") " pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.809888 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.809865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:02.950586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:02.950558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-d66b8"]
Jul 01 18:48:02.953960 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:48:02.953924    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d47e275_d2e7_470a_8a65_de73969ee068.slice/crio-3507e45398b38154b32a0190f17318d4ec3756eefe123a6e07061cb32feca0b9 WatchSource:0}: Error finding container 3507e45398b38154b32a0190f17318d4ec3756eefe123a6e07061cb32feca0b9: Status 404 returned error can't find the container with id 3507e45398b38154b32a0190f17318d4ec3756eefe123a6e07061cb32feca0b9
Jul 01 18:48:03.899858 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:03.899817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d66b8" event={"ID":"3d47e275-d2e7-470a-8a65-de73969ee068","Type":"ContainerStarted","Data":"33a6132ba36063175c6360a57f29edfa5a7e7ae18289de287c95e4269041e861"}
Jul 01 18:48:03.900342 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:03.900313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d66b8" event={"ID":"3d47e275-d2e7-470a-8a65-de73969ee068","Type":"ContainerStarted","Data":"3507e45398b38154b32a0190f17318d4ec3756eefe123a6e07061cb32feca0b9"}
Jul 01 18:48:04.905217 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:04.905183    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d47e275-d2e7-470a-8a65-de73969ee068" containerID="33a6132ba36063175c6360a57f29edfa5a7e7ae18289de287c95e4269041e861" exitCode=0
Jul 01 18:48:04.905600 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:04.905258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d66b8" event={"ID":"3d47e275-d2e7-470a-8a65-de73969ee068","Type":"ContainerDied","Data":"33a6132ba36063175c6360a57f29edfa5a7e7ae18289de287c95e4269041e861"}
Jul 01 18:48:05.911591 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:05.911560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d66b8" event={"ID":"3d47e275-d2e7-470a-8a65-de73969ee068","Type":"ContainerStarted","Data":"4f79ca8d27004593df0d9c0db1ea71ab93a1709c4278856a0afb139921e8add3"}
Jul 01 18:48:05.931422 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:05.931365    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-d66b8" podStartSLOduration=2.25239765 podStartE2EDuration="3.931346686s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.678949036s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:48:05.931176755 +0000 UTC m=+644.166131021" watchObservedRunningTime="2026-07-01 18:48:05.931346686 +0000 UTC m=+644.166300954"
Jul 01 18:48:12.946898 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:12.946867    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d47e275-d2e7-470a-8a65-de73969ee068" containerID="4f79ca8d27004593df0d9c0db1ea71ab93a1709c4278856a0afb139921e8add3" exitCode=0
Jul 01 18:48:12.947380 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:12.946937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d66b8" event={"ID":"3d47e275-d2e7-470a-8a65-de73969ee068","Type":"ContainerDied","Data":"4f79ca8d27004593df0d9c0db1ea71ab93a1709c4278856a0afb139921e8add3"}
Jul 01 18:48:14.113018 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.112993    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:14.129446 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.129425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d47e275-d2e7-470a-8a65-de73969ee068-model-cache\") pod \"3d47e275-d2e7-470a-8a65-de73969ee068\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") "
Jul 01 18:48:14.129520 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.129453    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdjs2\" (UniqueName: \"kubernetes.io/projected/3d47e275-d2e7-470a-8a65-de73969ee068-kube-api-access-pdjs2\") pod \"3d47e275-d2e7-470a-8a65-de73969ee068\" (UID: \"3d47e275-d2e7-470a-8a65-de73969ee068\") "
Jul 01 18:48:14.131920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.131889    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d47e275-d2e7-470a-8a65-de73969ee068-kube-api-access-pdjs2" (OuterVolumeSpecName: "kube-api-access-pdjs2") pod "3d47e275-d2e7-470a-8a65-de73969ee068" (UID: "3d47e275-d2e7-470a-8a65-de73969ee068"). InnerVolumeSpecName "kube-api-access-pdjs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:48:14.196118 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.196082    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d47e275-d2e7-470a-8a65-de73969ee068-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d47e275-d2e7-470a-8a65-de73969ee068" (UID: "3d47e275-d2e7-470a-8a65-de73969ee068"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:48:14.229907 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.229886    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d47e275-d2e7-470a-8a65-de73969ee068-model-cache\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:48:14.230007 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.229925    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjs2\" (UniqueName: \"kubernetes.io/projected/3d47e275-d2e7-470a-8a65-de73969ee068-kube-api-access-pdjs2\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:48:14.956743 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.956708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-d66b8" event={"ID":"3d47e275-d2e7-470a-8a65-de73969ee068","Type":"ContainerDied","Data":"3507e45398b38154b32a0190f17318d4ec3756eefe123a6e07061cb32feca0b9"}
Jul 01 18:48:14.956922 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.956752    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3507e45398b38154b32a0190f17318d4ec3756eefe123a6e07061cb32feca0b9"
Jul 01 18:48:14.956922 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:14.956723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-d66b8"
Jul 01 18:48:15.339967 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:15.339934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:15.440908 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:15.440875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:31.536176 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.536128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:31.643891 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.643864    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"]
Jul 01 18:48:31.644194 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.644182    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d47e275-d2e7-470a-8a65-de73969ee068" containerName="s3-tls-init-serving"
Jul 01 18:48:31.644236 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.644195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d47e275-d2e7-470a-8a65-de73969ee068" containerName="s3-tls-init-serving"
Jul 01 18:48:31.644236 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.644213    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d47e275-d2e7-470a-8a65-de73969ee068" containerName="download-hf-model"
Jul 01 18:48:31.644236 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.644218    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d47e275-d2e7-470a-8a65-de73969ee068" containerName="download-hf-model"
Jul 01 18:48:31.644325 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.644284    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d47e275-d2e7-470a-8a65-de73969ee068" containerName="s3-tls-init-serving"
Jul 01 18:48:31.646572 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.646558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.651880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.651846    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:48:31.652009 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.651963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:48:31.652009 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.651969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 01 18:48:31.652123 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.652088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:48:31.653393 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.653003    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 01 18:48:31.653491 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.653470    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:48:31.653726 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.653711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6866r\""
Jul 01 18:48:31.673902 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.673878    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"]
Jul 01 18:48:31.771614 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.771577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4a119b-a48d-4bfb-bf74-3586582e210c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.771614 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.771615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scggr\" (UniqueName: \"kubernetes.io/projected/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-api-access-scggr\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.771898 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.771637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4a119b-a48d-4bfb-bf74-3586582e210c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.771898 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.771682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.771898 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.771739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4a119b-a48d-4bfb-bf74-3586582e210c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scggr\" (UniqueName: \"kubernetes.io/projected/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-api-access-scggr\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4a119b-a48d-4bfb-bf74-3586582e210c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873142 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873455 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873538 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4a119b-a48d-4bfb-bf74-3586582e210c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873930 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.873983 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.873910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.875612 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.875594    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4a119b-a48d-4bfb-bf74-3586582e210c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.883586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.883561    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scggr\" (UniqueName: \"kubernetes.io/projected/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-api-access-scggr\") pod \"isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:31.908492 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.908465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:31.956482 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:31.956451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:32.095514 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:32.095484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"]
Jul 01 18:48:32.098651 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:48:32.098624    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4a119b_a48d_4bfb_bf74_3586582e210c.slice/crio-13569b62d344a9e550ab81ea9287a1bf0a82ed1e56648da919880c7d79fac6ff WatchSource:0}: Error finding container 13569b62d344a9e550ab81ea9287a1bf0a82ed1e56648da919880c7d79fac6ff: Status 404 returned error can't find the container with id 13569b62d344a9e550ab81ea9287a1bf0a82ed1e56648da919880c7d79fac6ff
Jul 01 18:48:33.034065 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:33.034029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerStarted","Data":"5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246"}
Jul 01 18:48:33.034448 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:33.034118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerStarted","Data":"13569b62d344a9e550ab81ea9287a1bf0a82ed1e56648da919880c7d79fac6ff"}
Jul 01 18:48:36.048215 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:36.048134    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerID="5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246" exitCode=0
Jul 01 18:48:36.048215 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:36.048205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerDied","Data":"5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246"}
Jul 01 18:48:52.145823 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:52.145763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerStarted","Data":"bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d"}
Jul 01 18:48:54.157761 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:54.157725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerStarted","Data":"a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a"}
Jul 01 18:48:54.409213 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:54.409120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:54.507907 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:54.507876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:48:57.175017 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:57.174977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerStarted","Data":"e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281"}
Jul 01 18:48:57.175483 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:57.175073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:57.175483 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:57.175100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:57.176520 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:57.176496    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:48:57.198920 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:57.198875    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podStartSLOduration=2.189695637 podStartE2EDuration="26.198859832s" totalImagesPullingTime="20.403214478s" totalInitContainerRuntime="3.605949717s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:48:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:48:57.197496074 +0000 UTC m=+695.432450341" watchObservedRunningTime="2026-07-01 18:48:57.198859832 +0000 UTC m=+695.433814098"
Jul 01 18:48:58.180106 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:58.180074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:58.180570 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:58.180225    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:48:58.181302 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:58.181269    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:48:58.184178 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:58.184161    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:48:59.184761 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:59.184720    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:48:59.185182 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:48:59.185066    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:00.189795 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:00.189760    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:49:00.190259 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:00.190139    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:10.189963 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:10.189919    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:49:10.190436 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:10.190319    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:20.190745 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:20.190652    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:49:20.191151 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:20.191057    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:26.684853 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:26.684793    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:30.190080 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:30.190033    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:49:30.190493 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:30.190458    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:40.189967 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:40.189920    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:49:40.190541 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:40.190326    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:49:45.496471 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:45.496436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:49:50.190076 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:50.190033    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:49:50.190591 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:49:50.190568    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:00.190170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:00.190120    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:50:00.190546 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:00.190493    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:10.190439 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:10.190411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:10.190881 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:10.190546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:16.843276 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.843243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:50:16.913075 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.913049    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"]
Jul 01 18:50:16.913342 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.913309    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" containerID="cri-o://bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d" gracePeriod=30
Jul 01 18:50:16.913491 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.913333    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" containerID="cri-o://e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281" gracePeriod=30
Jul 01 18:50:16.913491 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.913372    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" containerID="cri-o://a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a" gracePeriod=30
Jul 01 18:50:16.924391 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.924367    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"]
Jul 01 18:50:16.927082 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.927066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:16.930014 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.929993    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 01 18:50:16.930116 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.930072    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 01 18:50:16.944617 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.944593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:50:16.947515 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.947495    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"]
Jul 01 18:50:16.977559 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.977539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:16.977686 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.977570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:16.977686 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.977593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4th\" (UniqueName: \"kubernetes.io/projected/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-api-access-mz4th\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:16.977686 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.977658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:16.977790 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:16.977725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.078637 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.078605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.078637 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.078642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.078897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.078664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4th\" (UniqueName: \"kubernetes.io/projected/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-api-access-mz4th\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.078897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.078705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.078897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.078738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.079199 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.079166    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.079342 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.079322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.079417 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.079400    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.081125 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.081101    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.096395 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.096345    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4th\" (UniqueName: \"kubernetes.io/projected/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-api-access-mz4th\") pod \"isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.169357 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.169327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:50:17.236618 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.236583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:17.391829 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:50:17.391775    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fa4dcc_6c3e_4dd7_a288_2aba0f33b6e5.slice/crio-cd4506a2ad3f061bb0d9a1e8be03f449dd16d4b7129e475e7e12780470b2c564 WatchSource:0}: Error finding container cd4506a2ad3f061bb0d9a1e8be03f449dd16d4b7129e475e7e12780470b2c564: Status 404 returned error can't find the container with id cd4506a2ad3f061bb0d9a1e8be03f449dd16d4b7129e475e7e12780470b2c564
Jul 01 18:50:17.391932 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.391838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"]
Jul 01 18:50:17.544092 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.544056    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerID="a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a" exitCode=2
Jul 01 18:50:17.544263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.544134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerDied","Data":"a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a"}
Jul 01 18:50:17.545710 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.545686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerStarted","Data":"4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83"}
Jul 01 18:50:17.545845 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:17.545747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerStarted","Data":"cd4506a2ad3f061bb0d9a1e8be03f449dd16d4b7129e475e7e12780470b2c564"}
Jul 01 18:50:18.181232 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:18.181192    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:50:20.189953 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:20.189912    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:50:20.190391 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:20.190235    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:21.565192 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:21.565157    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerID="bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d" exitCode=0
Jul 01 18:50:21.565621 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:21.565236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerDied","Data":"bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d"}
Jul 01 18:50:21.566708 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:21.566684    2577 generic.go:356] "Generic (PLEG): container finished" podID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerID="4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83" exitCode=0
Jul 01 18:50:21.566828 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:21.566732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerDied","Data":"4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83"}
Jul 01 18:50:22.574223 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:22.574189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerStarted","Data":"79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6"}
Jul 01 18:50:22.574616 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:22.574259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerStarted","Data":"75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707"}
Jul 01 18:50:22.574616 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:22.574285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerStarted","Data":"fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0"}
Jul 01 18:50:22.574616 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:22.574477    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:22.617146 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:22.617103    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podStartSLOduration=2.953867412 podStartE2EDuration="6.61708635s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663218938s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:50:22.613287407 +0000 UTC m=+780.848241693" watchObservedRunningTime="2026-07-01 18:50:22.61708635 +0000 UTC m=+780.852040615"
Jul 01 18:50:23.180980 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:23.180944    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:50:23.579007 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:23.578961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:23.579386 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:23.579044    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:23.580384 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:23.580356    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:50:23.580995 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:23.580971    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:24.583440 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:24.583406    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:50:24.583903 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:24.583835    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:28.181114 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:28.181067    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:50:28.181531 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:28.181203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:29.588097 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:29.588068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:50:29.588828 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:29.588758    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:50:29.589150 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:29.589122    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:30.189749 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:30.189704    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:50:30.190053 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:30.190028    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:33.180474 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:33.180427    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:50:38.180342 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:38.180300    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:50:39.589032 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:39.588990    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:50:39.589497 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:39.589446    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:40.189710 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:40.189665    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 18:50:40.189926 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:40.189846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:40.190120 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:40.190100    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:40.190192 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:40.190181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:43.180285 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:43.180243    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 18:50:47.070509 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.070476    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:47.244457 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-rbac-proxy-sar-config\") pod \"6f4a119b-a48d-4bfb-bf74-3586582e210c\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") "
Jul 01 18:50:47.244457 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244464    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scggr\" (UniqueName: \"kubernetes.io/projected/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-api-access-scggr\") pod \"6f4a119b-a48d-4bfb-bf74-3586582e210c\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") "
Jul 01 18:50:47.244664 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4a119b-a48d-4bfb-bf74-3586582e210c-proxy-tls\") pod \"6f4a119b-a48d-4bfb-bf74-3586582e210c\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") "
Jul 01 18:50:47.244664 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244578    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4a119b-a48d-4bfb-bf74-3586582e210c-kserve-provision-location\") pod \"6f4a119b-a48d-4bfb-bf74-3586582e210c\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") "
Jul 01 18:50:47.244664 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-cabundle-cert\") pod \"6f4a119b-a48d-4bfb-bf74-3586582e210c\" (UID: \"6f4a119b-a48d-4bfb-bf74-3586582e210c\") "
Jul 01 18:50:47.244970 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244937    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f4a119b-a48d-4bfb-bf74-3586582e210c" (UID: "6f4a119b-a48d-4bfb-bf74-3586582e210c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:50:47.245084 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.244969    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4a119b-a48d-4bfb-bf74-3586582e210c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f4a119b-a48d-4bfb-bf74-3586582e210c" (UID: "6f4a119b-a48d-4bfb-bf74-3586582e210c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:50:47.245084 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.245037    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f4a119b-a48d-4bfb-bf74-3586582e210c" (UID: "6f4a119b-a48d-4bfb-bf74-3586582e210c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:50:47.246830 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.246786    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4a119b-a48d-4bfb-bf74-3586582e210c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f4a119b-a48d-4bfb-bf74-3586582e210c" (UID: "6f4a119b-a48d-4bfb-bf74-3586582e210c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:50:47.246928 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.246797    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-api-access-scggr" (OuterVolumeSpecName: "kube-api-access-scggr") pod "6f4a119b-a48d-4bfb-bf74-3586582e210c" (UID: "6f4a119b-a48d-4bfb-bf74-3586582e210c"). InnerVolumeSpecName "kube-api-access-scggr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:50:47.345433 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.345386    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:50:47.345433 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.345430    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scggr\" (UniqueName: \"kubernetes.io/projected/6f4a119b-a48d-4bfb-bf74-3586582e210c-kube-api-access-scggr\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:50:47.345642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.345445    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f4a119b-a48d-4bfb-bf74-3586582e210c-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:50:47.345642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.345461    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f4a119b-a48d-4bfb-bf74-3586582e210c-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:50:47.345642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.345473    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f4a119b-a48d-4bfb-bf74-3586582e210c-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:50:47.696648 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.696555    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerID="e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281" exitCode=0
Jul 01 18:50:47.696648 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.696626    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"
Jul 01 18:50:47.696893 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.696644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerDied","Data":"e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281"}
Jul 01 18:50:47.696893 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.696722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f" event={"ID":"6f4a119b-a48d-4bfb-bf74-3586582e210c","Type":"ContainerDied","Data":"13569b62d344a9e550ab81ea9287a1bf0a82ed1e56648da919880c7d79fac6ff"}
Jul 01 18:50:47.696893 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.696754    2577 scope.go:117] "RemoveContainer" containerID="e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281"
Jul 01 18:50:47.705266 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.705251    2577 scope.go:117] "RemoveContainer" containerID="a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a"
Jul 01 18:50:47.712930 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.712914    2577 scope.go:117] "RemoveContainer" containerID="bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d"
Jul 01 18:50:47.719904 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.719888    2577 scope.go:117] "RemoveContainer" containerID="5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246"
Jul 01 18:50:47.723842 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.723817    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"]
Jul 01 18:50:47.727928 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.727911    2577 scope.go:117] "RemoveContainer" containerID="e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281"
Jul 01 18:50:47.728105 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728089    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-869f8b8c7d-4zp9f"]
Jul 01 18:50:47.728188 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:50:47.728164    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281\": container with ID starting with e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281 not found: ID does not exist" containerID="e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281"
Jul 01 18:50:47.728241 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728200    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281"} err="failed to get container status \"e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281\": rpc error: code = NotFound desc = could not find container \"e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281\": container with ID starting with e4e3364fc0157d430a7537219d57ad5f250e2a68cbdbfe2c540774d38c3a2281 not found: ID does not exist"
Jul 01 18:50:47.728241 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728221    2577 scope.go:117] "RemoveContainer" containerID="a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a"
Jul 01 18:50:47.728457 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:50:47.728443    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a\": container with ID starting with a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a not found: ID does not exist" containerID="a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a"
Jul 01 18:50:47.728490 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728460    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a"} err="failed to get container status \"a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a\": rpc error: code = NotFound desc = could not find container \"a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a\": container with ID starting with a70641dd9ed2c781468b79e9ca21eaef0eab2f5a95a70048e91521e1620ac05a not found: ID does not exist"
Jul 01 18:50:47.728490 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728474    2577 scope.go:117] "RemoveContainer" containerID="bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d"
Jul 01 18:50:47.728718 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:50:47.728699    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d\": container with ID starting with bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d not found: ID does not exist" containerID="bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d"
Jul 01 18:50:47.728835 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728722    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d"} err="failed to get container status \"bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d\": rpc error: code = NotFound desc = could not find container \"bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d\": container with ID starting with bc2343dca8d93f687716c440fe60b618b11b9cf38747caee994ddaf269e7c70d not found: ID does not exist"
Jul 01 18:50:47.728835 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.728737    2577 scope.go:117] "RemoveContainer" containerID="5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246"
Jul 01 18:50:47.729002 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:50:47.728987    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246\": container with ID starting with 5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246 not found: ID does not exist" containerID="5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246"
Jul 01 18:50:47.729046 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:47.729005    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246"} err="failed to get container status \"5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246\": rpc error: code = NotFound desc = could not find container \"5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246\": container with ID starting with 5df084fdb1c4b0fb01e88fc5b3c62dee69e66e8fb32a603ac9579ebf80c6d246 not found: ID does not exist"
Jul 01 18:50:48.207816 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:48.207772    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" path="/var/lib/kubelet/pods/6f4a119b-a48d-4bfb-bf74-3586582e210c/volumes"
Jul 01 18:50:49.589536 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:49.589491    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:50:49.590067 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:49.590044    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:50:59.589113 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:59.589067    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:50:59.589513 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:50:59.589458    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:51:09.588938 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:09.588886    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:51:09.589368 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:09.589353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:51:19.588671 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:19.588624    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:51:19.589235 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:19.589087    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:51:29.588671 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:29.588619    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:51:29.589118 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:29.589048    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:51:39.203566 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:39.203539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:51:39.203970 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:39.203750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:51:52.138165 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.138122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:51:52.169479 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169447    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"]
Jul 01 18:51:52.169830 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169800    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="storage-initializer"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169832    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="storage-initializer"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169850    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169855    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169863    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169868    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169883    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169887    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169942    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kube-rbac-proxy"
Jul 01 18:51:52.169954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169950    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="agent"
Jul 01 18:51:52.170222 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.169958    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4a119b-a48d-4bfb-bf74-3586582e210c" containerName="kserve-container"
Jul 01 18:51:52.172051 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.172036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.175069 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.175047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 01 18:51:52.175618 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.175601    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"]
Jul 01 18:51:52.175889 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.175862    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" containerID="cri-o://fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0" gracePeriod=30
Jul 01 18:51:52.175954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.175887    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 01 18:51:52.175954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.175920    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" containerID="cri-o://79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6" gracePeriod=30
Jul 01 18:51:52.176044 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.175936    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" containerID="cri-o://75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707" gracePeriod=30
Jul 01 18:51:52.189283 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.189237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"]
Jul 01 18:51:52.198453 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.198427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qt6m\" (UniqueName: \"kubernetes.io/projected/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-api-access-6qt6m\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.198571 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.198520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.198571 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.198546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.211252 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.211223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:51:52.299490 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.299458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.299660 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.299550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.299660 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.299612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt6m\" (UniqueName: \"kubernetes.io/projected/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-api-access-6qt6m\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.299781 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:51:52.299692    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 01 18:51:52.299781 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:51:52.299754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls podName:77f4c452-0a23-4e5d-8ac6-d94447733f15 nodeName:}" failed. No retries permitted until 2026-07-01 18:51:52.799732059 +0000 UTC m=+871.034686315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-b95c5" (UID: "77f4c452-0a23-4e5d-8ac6-d94447733f15") : secret "message-dumper-predictor-serving-cert" not found
Jul 01 18:51:52.300263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.300244    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.308485 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.308464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt6m\" (UniqueName: \"kubernetes.io/projected/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-api-access-6qt6m\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.588708 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.588677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:51:52.804201 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.804168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:52.806566 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:52.806542    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-b95c5\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:53.001275 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:53.001245    2577 generic.go:356] "Generic (PLEG): container finished" podID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerID="75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707" exitCode=2
Jul 01 18:51:53.001438 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:53.001316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerDied","Data":"75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707"}
Jul 01 18:51:53.083798 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:53.083765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:53.211389 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:53.211329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"]
Jul 01 18:51:53.214070 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:51:53.214046    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f4c452_0a23_4e5d_8ac6_d94447733f15.slice/crio-b3a571495f5e0409bb6f0cffccccda25f72222c524b9fd872c713081933c3e45 WatchSource:0}: Error finding container b3a571495f5e0409bb6f0cffccccda25f72222c524b9fd872c713081933c3e45: Status 404 returned error can't find the container with id b3a571495f5e0409bb6f0cffccccda25f72222c524b9fd872c713081933c3e45
Jul 01 18:51:53.216058 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:53.216038    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 18:51:54.006905 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:54.006865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" event={"ID":"77f4c452-0a23-4e5d-8ac6-d94447733f15","Type":"ContainerStarted","Data":"b3a571495f5e0409bb6f0cffccccda25f72222c524b9fd872c713081933c3e45"}
Jul 01 18:51:54.584020 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:54.583991    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:51:55.014519 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:55.014475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" event={"ID":"77f4c452-0a23-4e5d-8ac6-d94447733f15","Type":"ContainerStarted","Data":"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"}
Jul 01 18:51:55.014726 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:55.014651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:55.014726 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:55.014704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" event={"ID":"77f4c452-0a23-4e5d-8ac6-d94447733f15","Type":"ContainerStarted","Data":"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"}
Jul 01 18:51:55.014856 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:55.014740    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:55.015976 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:55.015953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:51:55.033544 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:55.033478    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" podStartSLOduration=1.715905293 podStartE2EDuration="3.033463428s" totalImagesPullingTime="1.317558135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:51:55.032625869 +0000 UTC m=+873.267580152" watchObservedRunningTime="2026-07-01 18:51:55.033463428 +0000 UTC m=+873.268417693"
Jul 01 18:51:57.029499 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:57.029469    2577 generic.go:356] "Generic (PLEG): container finished" podID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerID="fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0" exitCode=0
Jul 01 18:51:57.029899 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:57.029543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerDied","Data":"fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0"}
Jul 01 18:51:59.203767 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:59.203722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:51:59.204162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:59.204101    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:51:59.583874 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:51:59.583831    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:52:02.034273 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:02.034246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:52:04.583873 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:04.583836    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:52:04.584250 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:04.583945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:52:09.203544 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:09.203503    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:52:09.204015 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:09.203832    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:09.584164 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:09.584129    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:52:12.169379 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.169334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:52:12.234205 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.234171    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"]
Jul 01 18:52:12.237019 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.237004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.239739 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.239722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 01 18:52:12.239854 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.239839    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 01 18:52:12.253033 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.253010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"]
Jul 01 18:52:12.369884 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.369853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22829bf-f48b-433d-8234-34e8a2b74cf3-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.370060 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.369889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8fd\" (UniqueName: \"kubernetes.io/projected/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-api-access-fx8fd\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.370060 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.369917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.370060 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.369940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.370060 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.370016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.471243 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.471214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.471386 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.471293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22829bf-f48b-433d-8234-34e8a2b74cf3-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.471386 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.471328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8fd\" (UniqueName: \"kubernetes.io/projected/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-api-access-fx8fd\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.471386 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.471370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.471527 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.471403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.471575 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:52:12.471523    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 01 18:52:12.471625 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:52:12.471587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls podName:c22829bf-f48b-433d-8234-34e8a2b74cf3 nodeName:}" failed. No retries permitted until 2026-07-01 18:52:12.971564043 +0000 UTC m=+891.206518290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls") pod "isvc-logger-predictor-599b89d7c-5s764" (UID: "c22829bf-f48b-433d-8234-34e8a2b74cf3") : secret "isvc-logger-predictor-serving-cert" not found
Jul 01 18:52:12.471847 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.471826    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22829bf-f48b-433d-8234-34e8a2b74cf3-kserve-provision-location\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.472036 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.472018    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-cabundle-cert\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.472144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.472125    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.481113 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.481089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8fd\" (UniqueName: \"kubernetes.io/projected/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-api-access-fx8fd\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.671284 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.671251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:52:12.976357 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.976316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:12.978837 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:12.978797    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls\") pod \"isvc-logger-predictor-599b89d7c-5s764\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:13.167689 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:13.167659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:13.306922 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:13.306902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"]
Jul 01 18:52:13.309333 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:52:13.309304    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22829bf_f48b_433d_8234_34e8a2b74cf3.slice/crio-f4e198a6fa59b049a7586a5abbfdecb810109658bfcb921442c1103192737bfe WatchSource:0}: Error finding container f4e198a6fa59b049a7586a5abbfdecb810109658bfcb921442c1103192737bfe: Status 404 returned error can't find the container with id f4e198a6fa59b049a7586a5abbfdecb810109658bfcb921442c1103192737bfe
Jul 01 18:52:14.110516 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:14.110483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerStarted","Data":"9c25d5feb552a95c4079f52378bc35308f57f6dd43dc3845b54a7c4a4aae8c47"}
Jul 01 18:52:14.110706 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:14.110543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerStarted","Data":"f4e198a6fa59b049a7586a5abbfdecb810109658bfcb921442c1103192737bfe"}
Jul 01 18:52:14.583528 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:14.583483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:52:18.132941 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:18.132908    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerID="9c25d5feb552a95c4079f52378bc35308f57f6dd43dc3845b54a7c4a4aae8c47" exitCode=0
Jul 01 18:52:18.133359 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:18.132977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerDied","Data":"9c25d5feb552a95c4079f52378bc35308f57f6dd43dc3845b54a7c4a4aae8c47"}
Jul 01 18:52:19.140548 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.140509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerStarted","Data":"30102cc3d3354a2cd0569aadf5b81911fba172ff13424e15febc032a09310fde"}
Jul 01 18:52:19.140959 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.140583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerStarted","Data":"f24d3e034b2ecbdbe49207854f24d4e9bdb871d9dec0cb874b3ec996242be954"}
Jul 01 18:52:19.140959 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.140611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerStarted","Data":"a8479b475356b90cbbd2c72bcf5079a1fa9b8e93c075130b6b9fb8f46e57139a"}
Jul 01 18:52:19.140959 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.140900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:19.141074 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.141047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:19.142265 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.142242    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:19.164880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.164823    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podStartSLOduration=3.398373801 podStartE2EDuration="7.164783003s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766409202s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:52:19.163511157 +0000 UTC m=+897.398465467" watchObservedRunningTime="2026-07-01 18:52:19.164783003 +0000 UTC m=+897.399737270"
Jul 01 18:52:19.202935 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.202904    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jul 01 18:52:19.203103 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.203028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:52:19.203279 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.203261    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:19.203354 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.203342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:52:19.583789 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:19.583750    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 01 18:52:20.149074 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:20.149034    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:20.149458 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:20.149075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:20.150152 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:20.150126    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:21.153799 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:21.153753    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:21.154380 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:21.154358    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:22.208391 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.208352    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:52:22.210132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.210110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:52:22.215507 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.215486    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:52:22.216684 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.216664    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:52:22.329443 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.329419    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:52:22.457323 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz4th\" (UniqueName: \"kubernetes.io/projected/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-api-access-mz4th\") pod \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") "
Jul 01 18:52:22.457323 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-cabundle-cert\") pod \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") "
Jul 01 18:52:22.457323 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kserve-provision-location\") pod \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") "
Jul 01 18:52:22.457586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457363    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-rbac-proxy-sar-config\") pod \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") "
Jul 01 18:52:22.457586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-proxy-tls\") pod \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\" (UID: \"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5\") "
Jul 01 18:52:22.457704 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457678    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" (UID: "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:52:22.457761 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457713    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" (UID: "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:52:22.457761 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.457719    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" (UID: "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:52:22.459339 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.459317    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-api-access-mz4th" (OuterVolumeSpecName: "kube-api-access-mz4th") pod "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" (UID: "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5"). InnerVolumeSpecName "kube-api-access-mz4th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:52:22.459447 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.459423    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" (UID: "83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:52:22.558226 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.558194    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:52:22.558226 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.558221    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:52:22.558226 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.558232    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:52:22.558460 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.558243    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:52:22.558460 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:22.558254    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4th\" (UniqueName: \"kubernetes.io/projected/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5-kube-api-access-mz4th\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:52:23.164651 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.164615    2577 generic.go:356] "Generic (PLEG): container finished" podID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerID="79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6" exitCode=0
Jul 01 18:52:23.164868 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.164681    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"
Jul 01 18:52:23.164868 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.164689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerDied","Data":"79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6"}
Jul 01 18:52:23.164868 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.164754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg" event={"ID":"83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5","Type":"ContainerDied","Data":"cd4506a2ad3f061bb0d9a1e8be03f449dd16d4b7129e475e7e12780470b2c564"}
Jul 01 18:52:23.164868 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.164784    2577 scope.go:117] "RemoveContainer" containerID="79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6"
Jul 01 18:52:23.173633 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.173616    2577 scope.go:117] "RemoveContainer" containerID="75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707"
Jul 01 18:52:23.181169 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.180975    2577 scope.go:117] "RemoveContainer" containerID="fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0"
Jul 01 18:52:23.188359 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.188343    2577 scope.go:117] "RemoveContainer" containerID="4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83"
Jul 01 18:52:23.191524 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.191503    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"]
Jul 01 18:52:23.196936 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.196916    2577 scope.go:117] "RemoveContainer" containerID="79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6"
Jul 01 18:52:23.197184 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:52:23.197160    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6\": container with ID starting with 79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6 not found: ID does not exist" containerID="79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6"
Jul 01 18:52:23.197241 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197197    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6"} err="failed to get container status \"79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6\": rpc error: code = NotFound desc = could not find container \"79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6\": container with ID starting with 79bf2e3d0bb07643acbf4b5108d35bc00e2039a3fb42a2ceb263b92f3c6a91b6 not found: ID does not exist"
Jul 01 18:52:23.197241 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197220    2577 scope.go:117] "RemoveContainer" containerID="75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707"
Jul 01 18:52:23.197458 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:52:23.197438    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707\": container with ID starting with 75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707 not found: ID does not exist" containerID="75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707"
Jul 01 18:52:23.197536 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197469    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707"} err="failed to get container status \"75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707\": rpc error: code = NotFound desc = could not find container \"75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707\": container with ID starting with 75a00274b51bd800810ce5c00162bb211f84f4918c4f7490694253d68553b707 not found: ID does not exist"
Jul 01 18:52:23.197536 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197490    2577 scope.go:117] "RemoveContainer" containerID="fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0"
Jul 01 18:52:23.197620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197558    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5fccccdbd5-qj6rg"]
Jul 01 18:52:23.197719 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:52:23.197703    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0\": container with ID starting with fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0 not found: ID does not exist" containerID="fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0"
Jul 01 18:52:23.197757 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197724    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0"} err="failed to get container status \"fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0\": rpc error: code = NotFound desc = could not find container \"fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0\": container with ID starting with fd4a7b85e080b7b265ebac97938852c4e8c4460464622f3257fc7a17ed1404d0 not found: ID does not exist"
Jul 01 18:52:23.197757 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.197742    2577 scope.go:117] "RemoveContainer" containerID="4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83"
Jul 01 18:52:23.198002 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:52:23.197985    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83\": container with ID starting with 4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83 not found: ID does not exist" containerID="4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83"
Jul 01 18:52:23.198051 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:23.198006    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83"} err="failed to get container status \"4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83\": rpc error: code = NotFound desc = could not find container \"4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83\": container with ID starting with 4cd4d75b00deded84d9543f75892d05ab465471851d6eb38ae6c4d4111bb8c83 not found: ID does not exist"
Jul 01 18:52:24.213234 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:24.213201    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" path="/var/lib/kubelet/pods/83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5/volumes"
Jul 01 18:52:26.158872 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:26.158841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:52:26.159547 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:26.159513    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:26.159785 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:26.159751    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:36.159584 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:36.159542    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:36.161969 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:36.160003    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:46.160291 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:46.160245    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:46.160698 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:46.160666    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:52:56.159491 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:56.159454    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:52:56.159998 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:52:56.159855    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:06.160221 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:06.160177    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:53:06.160621 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:06.160559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:16.160173 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:16.160132    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:53:16.160636 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:16.160615    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:26.160356 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:26.160298    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:53:26.160869 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:26.160777    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:53:36.159884 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:36.159853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:53:36.160357 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:36.160139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:53:47.216729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.216654    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-b95c5_77f4c452-0a23-4e5d-8ac6-d94447733f15/kserve-container/0.log"
Jul 01 18:53:47.505425 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.505393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:53:47.588865 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.588831    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"]
Jul 01 18:53:47.589284 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.589248    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" containerID="cri-o://a8479b475356b90cbbd2c72bcf5079a1fa9b8e93c075130b6b9fb8f46e57139a" gracePeriod=30
Jul 01 18:53:47.589433 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.589299    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" containerID="cri-o://f24d3e034b2ecbdbe49207854f24d4e9bdb871d9dec0cb874b3ec996242be954" gracePeriod=30
Jul 01 18:53:47.589433 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.589301    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" containerID="cri-o://30102cc3d3354a2cd0569aadf5b81911fba172ff13424e15febc032a09310fde" gracePeriod=30
Jul 01 18:53:47.591067 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591046    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"]
Jul 01 18:53:47.591511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591495    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="storage-initializer"
Jul 01 18:53:47.591511 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591512    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="storage-initializer"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591521    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591526    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591539    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591545    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591555    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591609    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kube-rbac-proxy"
Jul 01 18:53:47.591622 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591618    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="kserve-container"
Jul 01 18:53:47.591912 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.591627    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fa4dcc-6c3e-4dd7-a288-2aba0f33b6e5" containerName="agent"
Jul 01 18:53:47.596515 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.596496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.598688 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.598669    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 01 18:53:47.600177 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.600163    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 01 18:53:47.612049 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.612027    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"]
Jul 01 18:53:47.647443 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.647416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.647586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.647466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjxt\" (UniqueName: \"kubernetes.io/projected/960fca5b-b33c-4533-ab4c-92df8927082b-kube-api-access-wpjxt\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.647586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.647484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.647586 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.647534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.647728 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.647587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960fca5b-b33c-4533-ab4c-92df8927082b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.665619 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.665589    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"]
Jul 01 18:53:47.665824 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.665781    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kserve-container" containerID="cri-o://48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6" gracePeriod=30
Jul 01 18:53:47.665885 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.665831    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kube-rbac-proxy" containerID="cri-o://a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c" gracePeriod=30
Jul 01 18:53:47.748147 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748282 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjxt\" (UniqueName: \"kubernetes.io/projected/960fca5b-b33c-4533-ab4c-92df8927082b-kube-api-access-wpjxt\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748282 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748282 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748282 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960fca5b-b33c-4533-ab4c-92df8927082b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748494 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:53:47.748354    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 01 18:53:47.748494 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:53:47.748410    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls podName:960fca5b-b33c-4533-ab4c-92df8927082b nodeName:}" failed. No retries permitted until 2026-07-01 18:53:48.248394895 +0000 UTC m=+986.483349140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls") pod "isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" (UID: "960fca5b-b33c-4533-ab4c-92df8927082b") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 01 18:53:47.748602 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748568    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960fca5b-b33c-4533-ab4c-92df8927082b-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748787 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748766    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.748897 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.748851    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.758229 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.758178    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjxt\" (UniqueName: \"kubernetes.io/projected/960fca5b-b33c-4533-ab4c-92df8927082b-kube-api-access-wpjxt\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:47.903303 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.903275    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:53:47.949506 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.949478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qt6m\" (UniqueName: \"kubernetes.io/projected/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-api-access-6qt6m\") pod \"77f4c452-0a23-4e5d-8ac6-d94447733f15\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") "
Jul 01 18:53:47.949678 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.949518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls\") pod \"77f4c452-0a23-4e5d-8ac6-d94447733f15\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") "
Jul 01 18:53:47.949678 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.949548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-rbac-proxy-sar-config\") pod \"77f4c452-0a23-4e5d-8ac6-d94447733f15\" (UID: \"77f4c452-0a23-4e5d-8ac6-d94447733f15\") "
Jul 01 18:53:47.949960 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.949938    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77f4c452-0a23-4e5d-8ac6-d94447733f15" (UID: "77f4c452-0a23-4e5d-8ac6-d94447733f15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:53:47.951656 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.951633    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77f4c452-0a23-4e5d-8ac6-d94447733f15" (UID: "77f4c452-0a23-4e5d-8ac6-d94447733f15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:53:47.951656 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:47.951639    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-api-access-6qt6m" (OuterVolumeSpecName: "kube-api-access-6qt6m") pod "77f4c452-0a23-4e5d-8ac6-d94447733f15" (UID: "77f4c452-0a23-4e5d-8ac6-d94447733f15"). InnerVolumeSpecName "kube-api-access-6qt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:53:48.050289 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.050211    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qt6m\" (UniqueName: \"kubernetes.io/projected/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-api-access-6qt6m\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:53:48.050289 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.050239    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77f4c452-0a23-4e5d-8ac6-d94447733f15-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:53:48.050289 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.050250    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77f4c452-0a23-4e5d-8ac6-d94447733f15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:53:48.252023 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.251982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:48.254263 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.254239    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-486qs\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:48.519091 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.519053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:53:48.564881 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.564846    2577 generic.go:356] "Generic (PLEG): container finished" podID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerID="a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c" exitCode=2
Jul 01 18:53:48.564881 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.564874    2577 generic.go:356] "Generic (PLEG): container finished" podID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerID="48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6" exitCode=2
Jul 01 18:53:48.564881 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.564879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" event={"ID":"77f4c452-0a23-4e5d-8ac6-d94447733f15","Type":"ContainerDied","Data":"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"}
Jul 01 18:53:48.565162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.564918    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"
Jul 01 18:53:48.565162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.564957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" event={"ID":"77f4c452-0a23-4e5d-8ac6-d94447733f15","Type":"ContainerDied","Data":"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"}
Jul 01 18:53:48.565162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.564987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5" event={"ID":"77f4c452-0a23-4e5d-8ac6-d94447733f15","Type":"ContainerDied","Data":"b3a571495f5e0409bb6f0cffccccda25f72222c524b9fd872c713081933c3e45"}
Jul 01 18:53:48.565162 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.565016    2577 scope.go:117] "RemoveContainer" containerID="a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"
Jul 01 18:53:48.569637 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.569613    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerID="f24d3e034b2ecbdbe49207854f24d4e9bdb871d9dec0cb874b3ec996242be954" exitCode=2
Jul 01 18:53:48.569740 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.569658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerDied","Data":"f24d3e034b2ecbdbe49207854f24d4e9bdb871d9dec0cb874b3ec996242be954"}
Jul 01 18:53:48.578456 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.578425    2577 scope.go:117] "RemoveContainer" containerID="48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"
Jul 01 18:53:48.588374 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.588280    2577 scope.go:117] "RemoveContainer" containerID="a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"
Jul 01 18:53:48.588772 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:53:48.588752    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c\": container with ID starting with a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c not found: ID does not exist" containerID="a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"
Jul 01 18:53:48.588880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.588775    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"} err="failed to get container status \"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c\": rpc error: code = NotFound desc = could not find container \"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c\": container with ID starting with a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c not found: ID does not exist"
Jul 01 18:53:48.588880 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.588797    2577 scope.go:117] "RemoveContainer" containerID="48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"
Jul 01 18:53:48.589076 ip-10-0-133-57 kubenswrapper[2577]: E0701 18:53:48.589059    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6\": container with ID starting with 48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6 not found: ID does not exist" containerID="48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"
Jul 01 18:53:48.589132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.589079    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"} err="failed to get container status \"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6\": rpc error: code = NotFound desc = could not find container \"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6\": container with ID starting with 48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6 not found: ID does not exist"
Jul 01 18:53:48.589132 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.589094    2577 scope.go:117] "RemoveContainer" containerID="a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"
Jul 01 18:53:48.589332 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.589311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c"} err="failed to get container status \"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c\": rpc error: code = NotFound desc = could not find container \"a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c\": container with ID starting with a1936173bc0f6ca23e42ed246a7876ded9e04c3c72b8631e548ff62ba82a257c not found: ID does not exist"
Jul 01 18:53:48.589405 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.589335    2577 scope.go:117] "RemoveContainer" containerID="48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"
Jul 01 18:53:48.589547 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.589530    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6"} err="failed to get container status \"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6\": rpc error: code = NotFound desc = could not find container \"48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6\": container with ID starting with 48749ca58d40b081fc03cdfebfbd568039af8833da073f2f15700b1f87b45db6 not found: ID does not exist"
Jul 01 18:53:48.589868 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.589853    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"]
Jul 01 18:53:48.591679 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.591658    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-b95c5"]
Jul 01 18:53:48.659460 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.659159    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"]
Jul 01 18:53:48.661957 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:53:48.661929    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960fca5b_b33c_4533_ab4c_92df8927082b.slice/crio-8d00fba08865d6c67cff5a0fd3f890d3523332660f1eba05d0978a12abb9668c WatchSource:0}: Error finding container 8d00fba08865d6c67cff5a0fd3f890d3523332660f1eba05d0978a12abb9668c: Status 404 returned error can't find the container with id 8d00fba08865d6c67cff5a0fd3f890d3523332660f1eba05d0978a12abb9668c
Jul 01 18:53:48.672828 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:48.672788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:53:49.576985 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:49.576955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerStarted","Data":"fac8fbab4eb3235fb3f22ab0005279642d0f9099c997244b34341be95c35451c"}
Jul 01 18:53:49.577409 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:49.577019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerStarted","Data":"8d00fba08865d6c67cff5a0fd3f890d3523332660f1eba05d0978a12abb9668c"}
Jul 01 18:53:50.209027 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:50.208997    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" path="/var/lib/kubelet/pods/77f4c452-0a23-4e5d-8ac6-d94447733f15/volumes"
Jul 01 18:53:51.155015 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:51.154962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:53:52.598399 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:52.598366    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerID="a8479b475356b90cbbd2c72bcf5079a1fa9b8e93c075130b6b9fb8f46e57139a" exitCode=0
Jul 01 18:53:52.598747 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:52.598439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerDied","Data":"a8479b475356b90cbbd2c72bcf5079a1fa9b8e93c075130b6b9fb8f46e57139a"}
Jul 01 18:53:53.604532 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:53.604500    2577 generic.go:356] "Generic (PLEG): container finished" podID="960fca5b-b33c-4533-ab4c-92df8927082b" containerID="fac8fbab4eb3235fb3f22ab0005279642d0f9099c997244b34341be95c35451c" exitCode=0
Jul 01 18:53:53.604948 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:53.604543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerDied","Data":"fac8fbab4eb3235fb3f22ab0005279642d0f9099c997244b34341be95c35451c"}
Jul 01 18:53:56.154531 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:56.154482    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:53:56.159893 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:56.159862    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:53:56.160180 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:53:56.160157    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:01.154321 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:01.154282    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:54:01.154727 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:01.154431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:54:06.154502 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:06.154455    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:54:06.159881 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:06.159836    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:54:06.160209 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:06.160159    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:11.154141 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:11.154099    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:54:16.154642 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:16.154606    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 01 18:54:16.160176 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:16.160073    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 01 18:54:16.160360 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:16.160258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:54:16.160932 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:16.160903    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 18:54:16.161056 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:16.161032    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:54:17.735036 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:17.734976    2577 generic.go:356] "Generic (PLEG): container finished" podID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerID="30102cc3d3354a2cd0569aadf5b81911fba172ff13424e15febc032a09310fde" exitCode=0
Jul 01 18:54:17.735427 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:17.735029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerDied","Data":"30102cc3d3354a2cd0569aadf5b81911fba172ff13424e15febc032a09310fde"}
Jul 01 18:54:18.247605 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.247583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:54:18.311706 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.311678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-rbac-proxy-sar-config\") pod \"c22829bf-f48b-433d-8234-34e8a2b74cf3\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") "
Jul 01 18:54:18.311930 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.311736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls\") pod \"c22829bf-f48b-433d-8234-34e8a2b74cf3\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") "
Jul 01 18:54:18.312006 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.311927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx8fd\" (UniqueName: \"kubernetes.io/projected/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-api-access-fx8fd\") pod \"c22829bf-f48b-433d-8234-34e8a2b74cf3\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") "
Jul 01 18:54:18.312066 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.312006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-cabundle-cert\") pod \"c22829bf-f48b-433d-8234-34e8a2b74cf3\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") "
Jul 01 18:54:18.312125 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.312090    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22829bf-f48b-433d-8234-34e8a2b74cf3-kserve-provision-location\") pod \"c22829bf-f48b-433d-8234-34e8a2b74cf3\" (UID: \"c22829bf-f48b-433d-8234-34e8a2b74cf3\") "
Jul 01 18:54:18.312125 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.312100    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c22829bf-f48b-433d-8234-34e8a2b74cf3" (UID: "c22829bf-f48b-433d-8234-34e8a2b74cf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:54:18.312416 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.312390    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c22829bf-f48b-433d-8234-34e8a2b74cf3" (UID: "c22829bf-f48b-433d-8234-34e8a2b74cf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:54:18.312537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.312434    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:54:18.312537 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.312467    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c22829bf-f48b-433d-8234-34e8a2b74cf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c22829bf-f48b-433d-8234-34e8a2b74cf3" (UID: "c22829bf-f48b-433d-8234-34e8a2b74cf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:54:18.314421 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.314391    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c22829bf-f48b-433d-8234-34e8a2b74cf3" (UID: "c22829bf-f48b-433d-8234-34e8a2b74cf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:54:18.314516 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.314423    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-api-access-fx8fd" (OuterVolumeSpecName: "kube-api-access-fx8fd") pod "c22829bf-f48b-433d-8234-34e8a2b74cf3" (UID: "c22829bf-f48b-433d-8234-34e8a2b74cf3"). InnerVolumeSpecName "kube-api-access-fx8fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:54:18.413629 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.413522    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8fd\" (UniqueName: \"kubernetes.io/projected/c22829bf-f48b-433d-8234-34e8a2b74cf3-kube-api-access-fx8fd\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:54:18.413629 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.413555    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c22829bf-f48b-433d-8234-34e8a2b74cf3-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:54:18.413629 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.413577    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c22829bf-f48b-433d-8234-34e8a2b74cf3-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:54:18.413629 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.413591    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c22829bf-f48b-433d-8234-34e8a2b74cf3-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:54:18.743181 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.743143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764" event={"ID":"c22829bf-f48b-433d-8234-34e8a2b74cf3","Type":"ContainerDied","Data":"f4e198a6fa59b049a7586a5abbfdecb810109658bfcb921442c1103192737bfe"}
Jul 01 18:54:18.743631 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.743184    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"
Jul 01 18:54:18.743631 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.743242    2577 scope.go:117] "RemoveContainer" containerID="30102cc3d3354a2cd0569aadf5b81911fba172ff13424e15febc032a09310fde"
Jul 01 18:54:18.753281 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.753242    2577 scope.go:117] "RemoveContainer" containerID="f24d3e034b2ecbdbe49207854f24d4e9bdb871d9dec0cb874b3ec996242be954"
Jul 01 18:54:18.762447 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.762431    2577 scope.go:117] "RemoveContainer" containerID="a8479b475356b90cbbd2c72bcf5079a1fa9b8e93c075130b6b9fb8f46e57139a"
Jul 01 18:54:18.771789 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.771767    2577 scope.go:117] "RemoveContainer" containerID="9c25d5feb552a95c4079f52378bc35308f57f6dd43dc3845b54a7c4a4aae8c47"
Jul 01 18:54:18.772061 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.772038    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"]
Jul 01 18:54:18.776020 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:18.776000    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-599b89d7c-5s764"]
Jul 01 18:54:20.215144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:54:20.215081    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" path="/var/lib/kubelet/pods/c22829bf-f48b-433d-8234-34e8a2b74cf3/volumes"
Jul 01 18:55:07.566403 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:55:07.566362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:56:17.368125 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:17.368089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerStarted","Data":"2d263e9d567cd823c461dee1ac0fd1294f0fc969d1312046c78bec7151e7a6ef"}
Jul 01 18:56:17.368652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:17.368196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:56:17.368652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:17.368231    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:56:17.368652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:17.368261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerStarted","Data":"35f6326591452f22817ef69b7066d50dbdab8ff1ba7b144da590d1c289019200"}
Jul 01 18:56:17.369718 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:17.369693    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:56:17.399583 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:17.399526    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podStartSLOduration=2.534074286 podStartE2EDuration="2m30.399510167s" totalImagesPullingTime="2m23.08655077s" totalInitContainerRuntime="4.778885111s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:56:17.398240673 +0000 UTC m=+1135.633194964" watchObservedRunningTime="2026-07-01 18:56:17.399510167 +0000 UTC m=+1135.634464432"
Jul 01 18:56:18.374081 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:18.374044    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:56:23.378652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:23.378627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:56:23.379194 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:23.379143    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 01 18:56:25.231370 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:25.231339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:56:25.315620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:25.315594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:56:33.379946 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:33.379921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:56:37.737073 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:37.737039    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"]
Jul 01 18:56:37.737514 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:37.737257    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container" containerID="cri-o://35f6326591452f22817ef69b7066d50dbdab8ff1ba7b144da590d1c289019200" gracePeriod=30
Jul 01 18:56:37.737514 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:37.737302    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kube-rbac-proxy" containerID="cri-o://2d263e9d567cd823c461dee1ac0fd1294f0fc969d1312046c78bec7151e7a6ef" gracePeriod=30
Jul 01 18:56:37.767312 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:37.767284    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:56:38.375082 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.375042    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 01 18:56:38.474901 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.474875    2577 generic.go:356] "Generic (PLEG): container finished" podID="960fca5b-b33c-4533-ab4c-92df8927082b" containerID="2d263e9d567cd823c461dee1ac0fd1294f0fc969d1312046c78bec7151e7a6ef" exitCode=2
Jul 01 18:56:38.474901 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.474899    2577 generic.go:356] "Generic (PLEG): container finished" podID="960fca5b-b33c-4533-ab4c-92df8927082b" containerID="35f6326591452f22817ef69b7066d50dbdab8ff1ba7b144da590d1c289019200" exitCode=0
Jul 01 18:56:38.475066 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.474948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerDied","Data":"2d263e9d567cd823c461dee1ac0fd1294f0fc969d1312046c78bec7151e7a6ef"}
Jul 01 18:56:38.475066 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.475014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerDied","Data":"35f6326591452f22817ef69b7066d50dbdab8ff1ba7b144da590d1c289019200"}
Jul 01 18:56:38.613770 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.613743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:56:38.771657 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.771625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960fca5b-b33c-4533-ab4c-92df8927082b-kserve-provision-location\") pod \"960fca5b-b33c-4533-ab4c-92df8927082b\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") "
Jul 01 18:56:38.772144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.771676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-cabundle-cert\") pod \"960fca5b-b33c-4533-ab4c-92df8927082b\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") "
Jul 01 18:56:38.772144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.771704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpjxt\" (UniqueName: \"kubernetes.io/projected/960fca5b-b33c-4533-ab4c-92df8927082b-kube-api-access-wpjxt\") pod \"960fca5b-b33c-4533-ab4c-92df8927082b\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") "
Jul 01 18:56:38.772144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.771741    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-kube-rbac-proxy-sar-config\") pod \"960fca5b-b33c-4533-ab4c-92df8927082b\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") "
Jul 01 18:56:38.772144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.771800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls\") pod \"960fca5b-b33c-4533-ab4c-92df8927082b\" (UID: \"960fca5b-b33c-4533-ab4c-92df8927082b\") "
Jul 01 18:56:38.772144 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.772129    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "960fca5b-b33c-4533-ab4c-92df8927082b" (UID: "960fca5b-b33c-4533-ab4c-92df8927082b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:56:38.772326 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.772152    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "960fca5b-b33c-4533-ab4c-92df8927082b" (UID: "960fca5b-b33c-4533-ab4c-92df8927082b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:56:38.773859 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.773833    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "960fca5b-b33c-4533-ab4c-92df8927082b" (UID: "960fca5b-b33c-4533-ab4c-92df8927082b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:56:38.774002 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.773981    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960fca5b-b33c-4533-ab4c-92df8927082b-kube-api-access-wpjxt" (OuterVolumeSpecName: "kube-api-access-wpjxt") pod "960fca5b-b33c-4533-ab4c-92df8927082b" (UID: "960fca5b-b33c-4533-ab4c-92df8927082b"). InnerVolumeSpecName "kube-api-access-wpjxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:56:38.774342 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.774322    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960fca5b-b33c-4533-ab4c-92df8927082b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "960fca5b-b33c-4533-ab4c-92df8927082b" (UID: "960fca5b-b33c-4533-ab4c-92df8927082b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:56:38.873168 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.873135    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:56:38.873168 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.873160    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/960fca5b-b33c-4533-ab4c-92df8927082b-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:56:38.873168 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.873171    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/960fca5b-b33c-4533-ab4c-92df8927082b-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:56:38.873392 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.873181    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/960fca5b-b33c-4533-ab4c-92df8927082b-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:56:38.873392 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:38.873192    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpjxt\" (UniqueName: \"kubernetes.io/projected/960fca5b-b33c-4533-ab4c-92df8927082b-kube-api-access-wpjxt\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:56:39.481307 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.481273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs" event={"ID":"960fca5b-b33c-4533-ab4c-92df8927082b","Type":"ContainerDied","Data":"8d00fba08865d6c67cff5a0fd3f890d3523332660f1eba05d0978a12abb9668c"}
Jul 01 18:56:39.481607 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.481339    2577 scope.go:117] "RemoveContainer" containerID="2d263e9d567cd823c461dee1ac0fd1294f0fc969d1312046c78bec7151e7a6ef"
Jul 01 18:56:39.481607 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.481277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"
Jul 01 18:56:39.490340 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.490324    2577 scope.go:117] "RemoveContainer" containerID="35f6326591452f22817ef69b7066d50dbdab8ff1ba7b144da590d1c289019200"
Jul 01 18:56:39.497843 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.497823    2577 scope.go:117] "RemoveContainer" containerID="fac8fbab4eb3235fb3f22ab0005279642d0f9099c997244b34341be95c35451c"
Jul 01 18:56:39.505123 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.505100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"]
Jul 01 18:56:39.509968 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:39.509948    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-486qs"]
Jul 01 18:56:40.208589 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:40.208558    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" path="/var/lib/kubelet/pods/960fca5b-b33c-4533-ab4c-92df8927082b/volumes"
Jul 01 18:56:42.973507 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:42.973470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:56:43.021360 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021329    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"]
Jul 01 18:56:43.021652 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021640    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021653    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021661    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="storage-initializer"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021668    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="storage-initializer"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021680    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kserve-container"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021685    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kserve-container"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021690    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="storage-initializer"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021696    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="storage-initializer"
Jul 01 18:56:43.021700 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021701    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021706    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021712    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021717    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021730    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021752    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021756    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021762    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021767    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021833    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021847    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kserve-container"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021860    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021868    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f4c452-0a23-4e5d-8ac6-d94447733f15" containerName="kube-rbac-proxy"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021873    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="agent"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021878    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="960fca5b-b33c-4533-ab4c-92df8927082b" containerName="kserve-container"
Jul 01 18:56:43.021996 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.021883    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22829bf-f48b-433d-8234-34e8a2b74cf3" containerName="kserve-container"
Jul 01 18:56:43.048844 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.048818    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"]
Jul 01 18:56:43.049009 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.048903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.052125 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.052102    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:56:43.052620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.052563    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 01 18:56:43.052620 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.052604    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:56:43.052819 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.052651    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:56:43.052819 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.052683    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6866r\""
Jul 01 18:56:43.053165 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.053140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:56:43.053274 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.053143    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 01 18:56:43.159212 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.159181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:56:43.211297 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.211260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzn56\" (UniqueName: \"kubernetes.io/projected/14760528-b504-4fd0-b411-9d839a905a30-kube-api-access-pzn56\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.211455 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.211311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14760528-b504-4fd0-b411-9d839a905a30-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.211455 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.211358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14760528-b504-4fd0-b411-9d839a905a30-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.211455 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.211423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.211601 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.211501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.312749 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.312664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.312749 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.312715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzn56\" (UniqueName: \"kubernetes.io/projected/14760528-b504-4fd0-b411-9d839a905a30-kube-api-access-pzn56\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.312749 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.312731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14760528-b504-4fd0-b411-9d839a905a30-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.313122 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.312767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14760528-b504-4fd0-b411-9d839a905a30-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.313122 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.312799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.313245 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.313210    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14760528-b504-4fd0-b411-9d839a905a30-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.313462 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.313442    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.313547 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.313523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.315215 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.315199    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14760528-b504-4fd0-b411-9d839a905a30-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.322013 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.321977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzn56\" (UniqueName: \"kubernetes.io/projected/14760528-b504-4fd0-b411-9d839a905a30-kube-api-access-pzn56\") pod \"isvc-autogluon-v2-predictor-67b849d456-jcl4q\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.359883 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.359846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:43.505438 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:43.505367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"]
Jul 01 18:56:43.508617 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:56:43.508596    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14760528_b504_4fd0_b411_9d839a905a30.slice/crio-f63f016e9023c28cfaebee6284649189f2c77c8f1e5a6884f30a3aa60dedb60c WatchSource:0}: Error finding container f63f016e9023c28cfaebee6284649189f2c77c8f1e5a6884f30a3aa60dedb60c: Status 404 returned error can't find the container with id f63f016e9023c28cfaebee6284649189f2c77c8f1e5a6884f30a3aa60dedb60c
Jul 01 18:56:44.511928 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:44.511892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerStarted","Data":"b26bc2b4f6dba6fbc5fe2accc5e08550b2bf536aef8e691b63ad5df6abae409e"}
Jul 01 18:56:44.512330 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:44.511948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerStarted","Data":"f63f016e9023c28cfaebee6284649189f2c77c8f1e5a6884f30a3aa60dedb60c"}
Jul 01 18:56:48.533245 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:48.533209    2577 generic.go:356] "Generic (PLEG): container finished" podID="14760528-b504-4fd0-b411-9d839a905a30" containerID="b26bc2b4f6dba6fbc5fe2accc5e08550b2bf536aef8e691b63ad5df6abae409e" exitCode=0
Jul 01 18:56:48.533683 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:48.533291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerDied","Data":"b26bc2b4f6dba6fbc5fe2accc5e08550b2bf536aef8e691b63ad5df6abae409e"}
Jul 01 18:56:49.541362 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:49.541330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerStarted","Data":"906d1008a4ce26401f3d370095803b571f1e8eb562d4ed76576f55afb3da2c26"}
Jul 01 18:56:49.541733 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:49.541388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerStarted","Data":"49735eff4387252a574bbe7e0cf11d081282b2e2a6947e2b475f96b801715e69"}
Jul 01 18:56:49.541733 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:49.541600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:49.541822 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:49.541757    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:56:49.567095 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:49.567044    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" podStartSLOduration=1.75143874 podStartE2EDuration="6.567027296s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.815588556s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:56:49.565222327 +0000 UTC m=+1167.800176626" watchObservedRunningTime="2026-07-01 18:56:49.567027296 +0000 UTC m=+1167.801981562"
Jul 01 18:56:55.553079 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:56:55.553051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:57:22.242729 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:57:22.242694    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:57:22.245460 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:57:22.245444    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 18:57:22.276525 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:57:22.276500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:57:22.276684 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:57:22.276501    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 18:58:16.131178 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:16.131098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:22.208574 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:22.208538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:58:23.180744 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.180714    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"]
Jul 01 18:58:23.181008 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.180984    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kserve-container" containerID="cri-o://49735eff4387252a574bbe7e0cf11d081282b2e2a6947e2b475f96b801715e69" gracePeriod=30
Jul 01 18:58:23.181123 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.181072    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kube-rbac-proxy" containerID="cri-o://906d1008a4ce26401f3d370095803b571f1e8eb562d4ed76576f55afb3da2c26" gracePeriod=30
Jul 01 18:58:23.218104 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.218080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:23.971309 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.971270    2577 generic.go:356] "Generic (PLEG): container finished" podID="14760528-b504-4fd0-b411-9d839a905a30" containerID="906d1008a4ce26401f3d370095803b571f1e8eb562d4ed76576f55afb3da2c26" exitCode=2
Jul 01 18:58:23.971309 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.971305    2577 generic.go:356] "Generic (PLEG): container finished" podID="14760528-b504-4fd0-b411-9d839a905a30" containerID="49735eff4387252a574bbe7e0cf11d081282b2e2a6947e2b475f96b801715e69" exitCode=0
Jul 01 18:58:23.971493 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.971361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerDied","Data":"906d1008a4ce26401f3d370095803b571f1e8eb562d4ed76576f55afb3da2c26"}
Jul 01 18:58:23.971493 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:23.971437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerDied","Data":"49735eff4387252a574bbe7e0cf11d081282b2e2a6947e2b475f96b801715e69"}
Jul 01 18:58:24.020760 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.020737    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:58:24.146736 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.146651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14760528-b504-4fd0-b411-9d839a905a30-kserve-provision-location\") pod \"14760528-b504-4fd0-b411-9d839a905a30\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") "
Jul 01 18:58:24.146736 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.146705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-cabundle-cert\") pod \"14760528-b504-4fd0-b411-9d839a905a30\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") "
Jul 01 18:58:24.146736 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.146735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-kube-rbac-proxy-sar-config\") pod \"14760528-b504-4fd0-b411-9d839a905a30\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") "
Jul 01 18:58:24.147014 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.146756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzn56\" (UniqueName: \"kubernetes.io/projected/14760528-b504-4fd0-b411-9d839a905a30-kube-api-access-pzn56\") pod \"14760528-b504-4fd0-b411-9d839a905a30\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") "
Jul 01 18:58:24.147014 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.146831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14760528-b504-4fd0-b411-9d839a905a30-proxy-tls\") pod \"14760528-b504-4fd0-b411-9d839a905a30\" (UID: \"14760528-b504-4fd0-b411-9d839a905a30\") "
Jul 01 18:58:24.147153 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.147134    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14760528-b504-4fd0-b411-9d839a905a30" (UID: "14760528-b504-4fd0-b411-9d839a905a30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:58:24.147193 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.147161    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14760528-b504-4fd0-b411-9d839a905a30" (UID: "14760528-b504-4fd0-b411-9d839a905a30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 18:58:24.148954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.148928    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14760528-b504-4fd0-b411-9d839a905a30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14760528-b504-4fd0-b411-9d839a905a30" (UID: "14760528-b504-4fd0-b411-9d839a905a30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 18:58:24.148954 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.148936    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14760528-b504-4fd0-b411-9d839a905a30-kube-api-access-pzn56" (OuterVolumeSpecName: "kube-api-access-pzn56") pod "14760528-b504-4fd0-b411-9d839a905a30" (UID: "14760528-b504-4fd0-b411-9d839a905a30"). InnerVolumeSpecName "kube-api-access-pzn56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 18:58:24.151228 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.151197    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14760528-b504-4fd0-b411-9d839a905a30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14760528-b504-4fd0-b411-9d839a905a30" (UID: "14760528-b504-4fd0-b411-9d839a905a30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 18:58:24.247911 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.247883    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14760528-b504-4fd0-b411-9d839a905a30-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:58:24.247911 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.247907    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:58:24.248238 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.247916    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14760528-b504-4fd0-b411-9d839a905a30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:58:24.248238 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.247925    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzn56\" (UniqueName: \"kubernetes.io/projected/14760528-b504-4fd0-b411-9d839a905a30-kube-api-access-pzn56\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:58:24.248238 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.247934    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14760528-b504-4fd0-b411-9d839a905a30-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 18:58:24.977878 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.977844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q" event={"ID":"14760528-b504-4fd0-b411-9d839a905a30","Type":"ContainerDied","Data":"f63f016e9023c28cfaebee6284649189f2c77c8f1e5a6884f30a3aa60dedb60c"}
Jul 01 18:58:24.978066 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.977919    2577 scope.go:117] "RemoveContainer" containerID="906d1008a4ce26401f3d370095803b571f1e8eb562d4ed76576f55afb3da2c26"
Jul 01 18:58:24.978066 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.977837    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"
Jul 01 18:58:24.988671 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.988653    2577 scope.go:117] "RemoveContainer" containerID="49735eff4387252a574bbe7e0cf11d081282b2e2a6947e2b475f96b801715e69"
Jul 01 18:58:24.996023 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:24.996007    2577 scope.go:117] "RemoveContainer" containerID="b26bc2b4f6dba6fbc5fe2accc5e08550b2bf536aef8e691b63ad5df6abae409e"
Jul 01 18:58:25.000549 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:25.000526    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"]
Jul 01 18:58:25.005000 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:25.004976    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-jcl4q"]
Jul 01 18:58:26.207849 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:26.207818    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14760528-b504-4fd0-b411-9d839a905a30" path="/var/lib/kubelet/pods/14760528-b504-4fd0-b411-9d839a905a30/volumes"
Jul 01 18:58:28.271288 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.271257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:28.352354 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352318    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"]
Jul 01 18:58:28.352660 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352648    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="storage-initializer"
Jul 01 18:58:28.352703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352661    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="storage-initializer"
Jul 01 18:58:28.352703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352671    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kserve-container"
Jul 01 18:58:28.352703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352677    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kserve-container"
Jul 01 18:58:28.352703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352684    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kube-rbac-proxy"
Jul 01 18:58:28.352703 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352690    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kube-rbac-proxy"
Jul 01 18:58:28.352904 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352737    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kube-rbac-proxy"
Jul 01 18:58:28.352904 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.352746    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14760528-b504-4fd0-b411-9d839a905a30" containerName="kserve-container"
Jul 01 18:58:28.359694 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.356926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.360635 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.360613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 01 18:58:28.360846 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.360636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 18:58:28.360846 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.360737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6866r\""
Jul 01 18:58:28.360846 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.360742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 18:58:28.361257 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.361227    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 18:58:28.361353 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.361265    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 01 18:58:28.361406 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.361366    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 18:58:28.367576 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.367556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"]
Jul 01 18:58:28.482887 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.482856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.482887 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.482890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b7879-7eee-47d4-acce-ad26a4c74060-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.483093 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.482911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.483093 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.482954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2tp\" (UniqueName: \"kubernetes.io/projected/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-api-access-hz2tp\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.483164 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.483106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b7879-7eee-47d4-acce-ad26a4c74060-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.583650 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.583564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b7879-7eee-47d4-acce-ad26a4c74060-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.583650 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.583607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.583650 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.583625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b7879-7eee-47d4-acce-ad26a4c74060-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.583650 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.583642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.583982 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.583662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2tp\" (UniqueName: \"kubernetes.io/projected/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-api-access-hz2tp\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.584082 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.584060    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b7879-7eee-47d4-acce-ad26a4c74060-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.584296 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.584278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.584378 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.584361    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.586090 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.586064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b7879-7eee-47d4-acce-ad26a4c74060-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.587187 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.587156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:28.594666 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.594639    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2tp\" (UniqueName: \"kubernetes.io/projected/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-api-access-hz2tp\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.669647 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.669604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:28.800134 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.800112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"]
Jul 01 18:58:28.802511 ip-10-0-133-57 kubenswrapper[2577]: W0701 18:58:28.802484    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72b7879_7eee_47d4_acce_ad26a4c74060.slice/crio-c8c8711d415b62a9fad83b1298251d6e47f58492496ba6a04d34954850ee162a WatchSource:0}: Error finding container c8c8711d415b62a9fad83b1298251d6e47f58492496ba6a04d34954850ee162a: Status 404 returned error can't find the container with id c8c8711d415b62a9fad83b1298251d6e47f58492496ba6a04d34954850ee162a
Jul 01 18:58:28.997230 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.997199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerStarted","Data":"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"}
Jul 01 18:58:28.997403 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:28.997257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerStarted","Data":"c8c8711d415b62a9fad83b1298251d6e47f58492496ba6a04d34954850ee162a"}
Jul 01 18:58:29.344549 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:29.344518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:29.408382 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:29.408328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 18:58:34.026491 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:34.026458    2577 generic.go:356] "Generic (PLEG): container finished" podID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerID="c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a" exitCode=0
Jul 01 18:58:34.026936 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:34.026516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerDied","Data":"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"}
Jul 01 18:58:35.032690 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:35.032660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerStarted","Data":"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"}
Jul 01 18:58:35.033170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:35.032716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerStarted","Data":"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"}
Jul 01 18:58:35.033170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:35.032936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:35.033170 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:35.033135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 18:58:35.058324 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:35.058277    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" podStartSLOduration=2.262587082 podStartE2EDuration="7.058262396s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.795675314s" isStatefulPod=false podCreationTimestamp="2026-07-01 18:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 18:58:35.056503675 +0000 UTC m=+1273.291457961" watchObservedRunningTime="2026-07-01 18:58:35.058262396 +0000 UTC m=+1273.293216658"
Jul 01 18:58:41.043696 ip-10-0-133-57 kubenswrapper[2577]: I0701 18:58:41.043666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 19:00:09.207557 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:09.207520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 19:00:18.474498 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:18.474461    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"]
Jul 01 19:00:18.475006 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:18.474758    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kserve-container" containerID="cri-o://c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a" gracePeriod=30
Jul 01 19:00:18.475006 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:18.474816    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kube-rbac-proxy" containerID="cri-o://20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf" gracePeriod=30
Jul 01 19:00:18.530512 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:18.530477    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:19.203857 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.203795    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-autogluon-v2-variants/ready\": dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 01 19:00:19.420139 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.420115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 19:00:19.506975 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.506942    2577 generic.go:356] "Generic (PLEG): container finished" podID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerID="20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf" exitCode=2
Jul 01 19:00:19.506975 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.506968    2577 generic.go:356] "Generic (PLEG): container finished" podID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerID="c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a" exitCode=0
Jul 01 19:00:19.507503 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.506995    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"
Jul 01 19:00:19.507503 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.507037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerDied","Data":"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"}
Jul 01 19:00:19.507503 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.507110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerDied","Data":"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"}
Jul 01 19:00:19.507503 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.507147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5" event={"ID":"c72b7879-7eee-47d4-acce-ad26a4c74060","Type":"ContainerDied","Data":"c8c8711d415b62a9fad83b1298251d6e47f58492496ba6a04d34954850ee162a"}
Jul 01 19:00:19.507503 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.507177    2577 scope.go:117] "RemoveContainer" containerID="20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"
Jul 01 19:00:19.512977 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.512959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b7879-7eee-47d4-acce-ad26a4c74060-kserve-provision-location\") pod \"c72b7879-7eee-47d4-acce-ad26a4c74060\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") "
Jul 01 19:00:19.513101 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.512997    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-rbac-proxy-sar-config\") pod \"c72b7879-7eee-47d4-acce-ad26a4c74060\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") "
Jul 01 19:00:19.513101 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.513023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b7879-7eee-47d4-acce-ad26a4c74060-proxy-tls\") pod \"c72b7879-7eee-47d4-acce-ad26a4c74060\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") "
Jul 01 19:00:19.513101 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.513054    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz2tp\" (UniqueName: \"kubernetes.io/projected/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-api-access-hz2tp\") pod \"c72b7879-7eee-47d4-acce-ad26a4c74060\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") "
Jul 01 19:00:19.513273 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.513105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-cabundle-cert\") pod \"c72b7879-7eee-47d4-acce-ad26a4c74060\" (UID: \"c72b7879-7eee-47d4-acce-ad26a4c74060\") "
Jul 01 19:00:19.513389 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.513364    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c72b7879-7eee-47d4-acce-ad26a4c74060" (UID: "c72b7879-7eee-47d4-acce-ad26a4c74060"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:00:19.513537 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.513513    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c72b7879-7eee-47d4-acce-ad26a4c74060" (UID: "c72b7879-7eee-47d4-acce-ad26a4c74060"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:00:19.515250 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.515225    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72b7879-7eee-47d4-acce-ad26a4c74060-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c72b7879-7eee-47d4-acce-ad26a4c74060" (UID: "c72b7879-7eee-47d4-acce-ad26a4c74060"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:00:19.515353 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.515246    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-api-access-hz2tp" (OuterVolumeSpecName: "kube-api-access-hz2tp") pod "c72b7879-7eee-47d4-acce-ad26a4c74060" (UID: "c72b7879-7eee-47d4-acce-ad26a4c74060"). InnerVolumeSpecName "kube-api-access-hz2tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:00:19.515475 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.515448    2577 scope.go:117] "RemoveContainer" containerID="c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"
Jul 01 19:00:19.518002 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.517977    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72b7879-7eee-47d4-acce-ad26a4c74060-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c72b7879-7eee-47d4-acce-ad26a4c74060" (UID: "c72b7879-7eee-47d4-acce-ad26a4c74060"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:00:19.530059 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.530041    2577 scope.go:117] "RemoveContainer" containerID="c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"
Jul 01 19:00:19.537570 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.537550    2577 scope.go:117] "RemoveContainer" containerID="20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"
Jul 01 19:00:19.537848 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:00:19.537831    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf\": container with ID starting with 20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf not found: ID does not exist" containerID="20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"
Jul 01 19:00:19.537917 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.537855    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"} err="failed to get container status \"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf\": rpc error: code = NotFound desc = could not find container \"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf\": container with ID starting with 20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf not found: ID does not exist"
Jul 01 19:00:19.537917 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.537872    2577 scope.go:117] "RemoveContainer" containerID="c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"
Jul 01 19:00:19.538126 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:00:19.538109    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a\": container with ID starting with c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a not found: ID does not exist" containerID="c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"
Jul 01 19:00:19.538169 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538135    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"} err="failed to get container status \"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a\": rpc error: code = NotFound desc = could not find container \"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a\": container with ID starting with c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a not found: ID does not exist"
Jul 01 19:00:19.538169 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538154    2577 scope.go:117] "RemoveContainer" containerID="c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"
Jul 01 19:00:19.538371 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:00:19.538356    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a\": container with ID starting with c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a not found: ID does not exist" containerID="c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"
Jul 01 19:00:19.538412 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538374    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"} err="failed to get container status \"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a\": rpc error: code = NotFound desc = could not find container \"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a\": container with ID starting with c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a not found: ID does not exist"
Jul 01 19:00:19.538412 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538390    2577 scope.go:117] "RemoveContainer" containerID="20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"
Jul 01 19:00:19.538582 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538564    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf"} err="failed to get container status \"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf\": rpc error: code = NotFound desc = could not find container \"20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf\": container with ID starting with 20528a3b0e04330b3b244e439f646f4d80b8cbc7359135000238fc6e7615d7cf not found: ID does not exist"
Jul 01 19:00:19.538621 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538584    2577 scope.go:117] "RemoveContainer" containerID="c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"
Jul 01 19:00:19.538790 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538772    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a"} err="failed to get container status \"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a\": rpc error: code = NotFound desc = could not find container \"c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a\": container with ID starting with c7d638582c68e16d5e7723e0362bb668edeb096a7f99d8eb86030fc85a7a8c6a not found: ID does not exist"
Jul 01 19:00:19.538873 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.538789    2577 scope.go:117] "RemoveContainer" containerID="c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"
Jul 01 19:00:19.539043 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.539027    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a"} err="failed to get container status \"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a\": rpc error: code = NotFound desc = could not find container \"c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a\": container with ID starting with c66b8aff80d7f7a53b7ceebd57b4ba1c05a1eed5d1b4982c9e0648a59730936a not found: ID does not exist"
Jul 01 19:00:19.613983 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.613950    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72b7879-7eee-47d4-acce-ad26a4c74060-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:00:19.613983 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.613976    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:00:19.613983 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.613986    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72b7879-7eee-47d4-acce-ad26a4c74060-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:00:19.614197 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.613999    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2tp\" (UniqueName: \"kubernetes.io/projected/c72b7879-7eee-47d4-acce-ad26a4c74060-kube-api-access-hz2tp\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:00:19.614197 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.614007    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72b7879-7eee-47d4-acce-ad26a4c74060-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:00:19.830485 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.830448    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"]
Jul 01 19:00:19.839790 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:19.839766    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-mgvp5"]
Jul 01 19:00:20.207648 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:20.207568    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" path="/var/lib/kubelet/pods/c72b7879-7eee-47d4-acce-ad26a4c74060/volumes"
Jul 01 19:00:23.591080 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.591044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:23.661197 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661164    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"]
Jul 01 19:00:23.661496 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661485    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kube-rbac-proxy"
Jul 01 19:00:23.661547 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661498    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kube-rbac-proxy"
Jul 01 19:00:23.661547 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661509    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kserve-container"
Jul 01 19:00:23.661547 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kserve-container"
Jul 01 19:00:23.661547 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="storage-initializer"
Jul 01 19:00:23.661547 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661535    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="storage-initializer"
Jul 01 19:00:23.661696 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661585    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kserve-container"
Jul 01 19:00:23.661696 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.661593    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72b7879-7eee-47d4-acce-ad26a4c74060" containerName="kube-rbac-proxy"
Jul 01 19:00:23.665751 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.665736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.668221 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.668200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 01 19:00:23.668340 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.668304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:00:23.668443 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.668426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:00:23.668537 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.668523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:00:23.669132 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.669112    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 01 19:00:23.669252 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.669144    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6866r\""
Jul 01 19:00:23.669252 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.669163    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:00:23.678310 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.678291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"]
Jul 01 19:00:23.752567 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.752536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnm7\" (UniqueName: \"kubernetes.io/projected/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-api-access-hcnm7\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.752738 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.752579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.752738 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.752616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.752738 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.752665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.752885 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.752731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854133 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854133 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854133 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854391 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnm7\" (UniqueName: \"kubernetes.io/projected/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-api-access-hcnm7\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854391 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854391 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:00:23.854289    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 01 19:00:23.854391 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:00:23.854359    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls podName:3346e75f-c9b5-4dc3-8978-0ddc804253f2 nodeName:}" failed. No retries permitted until 2026-07-01 19:00:24.354337846 +0000 UTC m=+1382.589292090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" (UID: "3346e75f-c9b5-4dc3-8978-0ddc804253f2") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 01 19:00:23.854563 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854467    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.854917 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.855022 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.854947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:23.863346 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:23.863324    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnm7\" (UniqueName: \"kubernetes.io/projected/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-api-access-hcnm7\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:24.013875 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:24.013846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:00:24.359456 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:24.359419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:24.361878 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:24.361855    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:24.576210 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:24.576177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:24.710276 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:24.710245    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"]
Jul 01 19:00:24.713589 ip-10-0-133-57 kubenswrapper[2577]: W0701 19:00:24.713564    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3346e75f_c9b5_4dc3_8978_0ddc804253f2.slice/crio-77fca1409653f80dd23966a10966597bd725d4bb64c9ced8a580e7a5b18f0ced WatchSource:0}: Error finding container 77fca1409653f80dd23966a10966597bd725d4bb64c9ced8a580e7a5b18f0ced: Status 404 returned error can't find the container with id 77fca1409653f80dd23966a10966597bd725d4bb64c9ced8a580e7a5b18f0ced
Jul 01 19:00:25.534510 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:25.534471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerStarted","Data":"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"}
Jul 01 19:00:25.534724 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:25.534533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerStarted","Data":"77fca1409653f80dd23966a10966597bd725d4bb64c9ced8a580e7a5b18f0ced"}
Jul 01 19:00:54.672531 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:54.672497    2577 generic.go:356] "Generic (PLEG): container finished" podID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerID="73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026" exitCode=0
Jul 01 19:00:54.672930 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:54.672538    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerDied","Data":"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"}
Jul 01 19:00:55.681068 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:55.681032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerStarted","Data":"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"}
Jul 01 19:00:55.681581 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:55.681099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerStarted","Data":"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"}
Jul 01 19:00:55.681581 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:55.681280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:00:55.701586 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:55.701534    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" podStartSLOduration=3.324942569 podStartE2EDuration="32.701519225s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.376576656s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:00:55.700304208 +0000 UTC m=+1413.935258508" watchObservedRunningTime="2026-07-01 19:00:55.701519225 +0000 UTC m=+1413.936473490"
Jul 01 19:00:56.685842 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:00:56.685814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:01:02.696976 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:01:02.696949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:02:22.306997 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:22.306969    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 19:02:22.308007 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:22.307988    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 19:02:22.313030 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:22.313004    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 19:02:22.313723 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:22.313701    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 19:02:33.205972 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:33.205933    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:02:33.772341 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:33.772296    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"]
Jul 01 19:02:34.132928 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:34.132795    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kserve-container" containerID="cri-o://7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5" gracePeriod=30
Jul 01 19:02:34.132928 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:34.132853    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kube-rbac-proxy" containerID="cri-o://ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d" gracePeriod=30
Jul 01 19:02:34.425145 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:34.425043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:02:35.076183 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.076157    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:02:35.093370 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-rbac-proxy-sar-config\") pod \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") "
Jul 01 19:02:35.093546 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kserve-provision-location\") pod \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") "
Jul 01 19:02:35.093546 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcnm7\" (UniqueName: \"kubernetes.io/projected/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-api-access-hcnm7\") pod \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") "
Jul 01 19:02:35.093546 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls\") pod \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") "
Jul 01 19:02:35.093546 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-cabundle-cert\") pod \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\" (UID: \"3346e75f-c9b5-4dc3-8978-0ddc804253f2\") "
Jul 01 19:02:35.093853 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093764    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3346e75f-c9b5-4dc3-8978-0ddc804253f2" (UID: "3346e75f-c9b5-4dc3-8978-0ddc804253f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:02:35.094000 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.093977    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3346e75f-c9b5-4dc3-8978-0ddc804253f2" (UID: "3346e75f-c9b5-4dc3-8978-0ddc804253f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:02:35.095551 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.095528    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-api-access-hcnm7" (OuterVolumeSpecName: "kube-api-access-hcnm7") pod "3346e75f-c9b5-4dc3-8978-0ddc804253f2" (UID: "3346e75f-c9b5-4dc3-8978-0ddc804253f2"). InnerVolumeSpecName "kube-api-access-hcnm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:02:35.096005 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.095981    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3346e75f-c9b5-4dc3-8978-0ddc804253f2" (UID: "3346e75f-c9b5-4dc3-8978-0ddc804253f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:02:35.098258 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.098233    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3346e75f-c9b5-4dc3-8978-0ddc804253f2" (UID: "3346e75f-c9b5-4dc3-8978-0ddc804253f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:02:35.139292 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139211    2577 generic.go:356] "Generic (PLEG): container finished" podID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerID="ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d" exitCode=2
Jul 01 19:02:35.139292 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139243    2577 generic.go:356] "Generic (PLEG): container finished" podID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerID="7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5" exitCode=0
Jul 01 19:02:35.139292 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139276    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"
Jul 01 19:02:35.139499 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerDied","Data":"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"}
Jul 01 19:02:35.139499 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerDied","Data":"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"}
Jul 01 19:02:35.139499 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg" event={"ID":"3346e75f-c9b5-4dc3-8978-0ddc804253f2","Type":"ContainerDied","Data":"77fca1409653f80dd23966a10966597bd725d4bb64c9ced8a580e7a5b18f0ced"}
Jul 01 19:02:35.139499 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.139449    2577 scope.go:117] "RemoveContainer" containerID="ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"
Jul 01 19:02:35.148389 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.148374    2577 scope.go:117] "RemoveContainer" containerID="7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"
Jul 01 19:02:35.156425 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.156403    2577 scope.go:117] "RemoveContainer" containerID="73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"
Jul 01 19:02:35.163773 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.163749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"]
Jul 01 19:02:35.164394 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.164379    2577 scope.go:117] "RemoveContainer" containerID="ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"
Jul 01 19:02:35.164714 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:02:35.164692    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d\": container with ID starting with ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d not found: ID does not exist" containerID="ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"
Jul 01 19:02:35.164761 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.164724    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"} err="failed to get container status \"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d\": rpc error: code = NotFound desc = could not find container \"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d\": container with ID starting with ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d not found: ID does not exist"
Jul 01 19:02:35.164761 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.164742    2577 scope.go:117] "RemoveContainer" containerID="7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"
Jul 01 19:02:35.165011 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:02:35.164991    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5\": container with ID starting with 7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5 not found: ID does not exist" containerID="7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"
Jul 01 19:02:35.165058 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165016    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"} err="failed to get container status \"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5\": rpc error: code = NotFound desc = could not find container \"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5\": container with ID starting with 7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5 not found: ID does not exist"
Jul 01 19:02:35.165058 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165034    2577 scope.go:117] "RemoveContainer" containerID="73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"
Jul 01 19:02:35.165245 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:02:35.165229    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026\": container with ID starting with 73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026 not found: ID does not exist" containerID="73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"
Jul 01 19:02:35.165313 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165252    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"} err="failed to get container status \"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026\": rpc error: code = NotFound desc = could not find container \"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026\": container with ID starting with 73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026 not found: ID does not exist"
Jul 01 19:02:35.165313 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165271    2577 scope.go:117] "RemoveContainer" containerID="ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"
Jul 01 19:02:35.165480 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165465    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d"} err="failed to get container status \"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d\": rpc error: code = NotFound desc = could not find container \"ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d\": container with ID starting with ca22640b873ec62ab32ec77b768ccd933baed09cc7efdd0fcc41cb2beaf26e3d not found: ID does not exist"
Jul 01 19:02:35.165530 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165480    2577 scope.go:117] "RemoveContainer" containerID="7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"
Jul 01 19:02:35.165714 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165691    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5"} err="failed to get container status \"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5\": rpc error: code = NotFound desc = could not find container \"7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5\": container with ID starting with 7576bf9c1717e0d130a683fde4a98c84b233173a63fe021b2962f38738f8f3c5 not found: ID does not exist"
Jul 01 19:02:35.165714 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165711    2577 scope.go:117] "RemoveContainer" containerID="73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"
Jul 01 19:02:35.165943 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.165925    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026"} err="failed to get container status \"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026\": rpc error: code = NotFound desc = could not find container \"73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026\": container with ID starting with 73831614c69100c689ddca0118bcec5e3ea7a504d3615b97e2196a014b5e7026 not found: ID does not exist"
Jul 01 19:02:35.175028 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.174994    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-g67fg"]
Jul 01 19:02:35.194643 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.194618    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:02:35.194643 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.194642    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:02:35.194775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.194653    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnm7\" (UniqueName: \"kubernetes.io/projected/3346e75f-c9b5-4dc3-8978-0ddc804253f2-kube-api-access-hcnm7\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:02:35.194775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.194662    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3346e75f-c9b5-4dc3-8978-0ddc804253f2-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:02:35.194775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:35.194672    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3346e75f-c9b5-4dc3-8978-0ddc804253f2-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:02:36.207871 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:36.207836    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" path="/var/lib/kubelet/pods/3346e75f-c9b5-4dc3-8978-0ddc804253f2/volumes"
Jul 01 19:02:38.874007 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.873973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:02:38.939988 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.939947    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"]
Jul 01 19:02:38.940352 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940335    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="storage-initializer"
Jul 01 19:02:38.940436 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940354    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="storage-initializer"
Jul 01 19:02:38.940436 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940370    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kube-rbac-proxy"
Jul 01 19:02:38.940436 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940377    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kube-rbac-proxy"
Jul 01 19:02:38.940436 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940408    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kserve-container"
Jul 01 19:02:38.940436 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940416    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kserve-container"
Jul 01 19:02:38.940693 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940498    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kube-rbac-proxy"
Jul 01 19:02:38.940693 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.940510    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3346e75f-c9b5-4dc3-8978-0ddc804253f2" containerName="kserve-container"
Jul 01 19:02:38.945422 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.945401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:38.947937 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.947909    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 01 19:02:38.948075 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.947955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:02:38.948075 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.947986    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:02:38.948975 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.948959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6866r\""
Jul 01 19:02:38.948975 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.948974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 19:02:38.949090 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.948958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 19:02:38.949090 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.948963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 01 19:02:38.956101 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:38.956080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"]
Jul 01 19:02:39.023586 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.023549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.023744 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.023606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.023744 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.023650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4shk\" (UniqueName: \"kubernetes.io/projected/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-api-access-t4shk\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.023744 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.023706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef1452a-b8b3-496a-8688-0d0d269a9414-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.023744 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.023725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125099 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef1452a-b8b3-496a-8688-0d0d269a9414-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125099 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125099 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125449 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125449 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4shk\" (UniqueName: \"kubernetes.io/projected/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-api-access-t4shk\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125449 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:02:39.125312    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 01 19:02:39.125449 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:02:39.125407    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls podName:bef1452a-b8b3-496a-8688-0d0d269a9414 nodeName:}" failed. No retries permitted until 2026-07-01 19:02:39.625385303 +0000 UTC m=+1517.860339548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" (UID: "bef1452a-b8b3-496a-8688-0d0d269a9414") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 01 19:02:39.125640 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125448    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef1452a-b8b3-496a-8688-0d0d269a9414-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125743 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125726    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.125779 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.125755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.136352 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.136327    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4shk\" (UniqueName: \"kubernetes.io/projected/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-api-access-t4shk\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.191136 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.191102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:02:39.628197 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.628155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.630587 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.630565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.856197 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.856161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:39.998228 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:39.998192    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"]
Jul 01 19:02:40.001712 ip-10-0-133-57 kubenswrapper[2577]: W0701 19:02:40.001676    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef1452a_b8b3_496a_8688_0d0d269a9414.slice/crio-17fb47d9422b6fc11e5c3ff38408fb8ee85e9e147ee1e762375cdad2a040e6ba WatchSource:0}: Error finding container 17fb47d9422b6fc11e5c3ff38408fb8ee85e9e147ee1e762375cdad2a040e6ba: Status 404 returned error can't find the container with id 17fb47d9422b6fc11e5c3ff38408fb8ee85e9e147ee1e762375cdad2a040e6ba
Jul 01 19:02:40.164205 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:40.164115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerStarted","Data":"c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f"}
Jul 01 19:02:40.164205 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:40.164185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerStarted","Data":"17fb47d9422b6fc11e5c3ff38408fb8ee85e9e147ee1e762375cdad2a040e6ba"}
Jul 01 19:02:48.716077 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:48.715989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:02:49.218879 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:49.218849    2577 generic.go:356] "Generic (PLEG): container finished" podID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerID="c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f" exitCode=0
Jul 01 19:02:49.219064 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:49.218919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerDied","Data":"c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f"}
Jul 01 19:02:50.225413 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:50.225383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerStarted","Data":"018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2"}
Jul 01 19:02:50.225821 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:50.225440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerStarted","Data":"79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8"}
Jul 01 19:02:50.225821 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:50.225629    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:50.247922 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:50.247867    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" podStartSLOduration=3.36017375 podStartE2EDuration="12.247852708s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.887678958s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:02:50.246602116 +0000 UTC m=+1528.481556382" watchObservedRunningTime="2026-07-01 19:02:50.247852708 +0000 UTC m=+1528.482806971"
Jul 01 19:02:51.230070 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:51.230039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:51.231442 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:51.231415    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:02:52.235989 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:52.235951    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:02:57.241247 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:57.241221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:02:57.241821 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:02:57.241777    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 01 19:03:07.242965 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:07.242936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:03:09.095033 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.094986    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"]
Jul 01 19:03:09.095489 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.095261    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container" containerID="cri-o://79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8" gracePeriod=30
Jul 01 19:03:09.095489 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.095292    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kube-rbac-proxy" containerID="cri-o://018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2" gracePeriod=30
Jul 01 19:03:09.163950 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.163906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:09.316483 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.316446    2577 generic.go:356] "Generic (PLEG): container finished" podID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerID="018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2" exitCode=2
Jul 01 19:03:09.316653 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.316514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerDied","Data":"018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2"}
Jul 01 19:03:09.945751 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:09.945726    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:03:10.098961 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.098869    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef1452a-b8b3-496a-8688-0d0d269a9414-kserve-provision-location\") pod \"bef1452a-b8b3-496a-8688-0d0d269a9414\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") "
Jul 01 19:03:10.098961 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.098912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-cabundle-cert\") pod \"bef1452a-b8b3-496a-8688-0d0d269a9414\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") "
Jul 01 19:03:10.098961 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.098946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls\") pod \"bef1452a-b8b3-496a-8688-0d0d269a9414\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") "
Jul 01 19:03:10.099509 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.098975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-rbac-proxy-sar-config\") pod \"bef1452a-b8b3-496a-8688-0d0d269a9414\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") "
Jul 01 19:03:10.099509 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.099000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4shk\" (UniqueName: \"kubernetes.io/projected/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-api-access-t4shk\") pod \"bef1452a-b8b3-496a-8688-0d0d269a9414\" (UID: \"bef1452a-b8b3-496a-8688-0d0d269a9414\") "
Jul 01 19:03:10.099509 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.099291    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bef1452a-b8b3-496a-8688-0d0d269a9414" (UID: "bef1452a-b8b3-496a-8688-0d0d269a9414"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:03:10.099509 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.099333    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bef1452a-b8b3-496a-8688-0d0d269a9414" (UID: "bef1452a-b8b3-496a-8688-0d0d269a9414"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:03:10.099950 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.099930    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef1452a-b8b3-496a-8688-0d0d269a9414-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bef1452a-b8b3-496a-8688-0d0d269a9414" (UID: "bef1452a-b8b3-496a-8688-0d0d269a9414"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:03:10.101158 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.101133    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-api-access-t4shk" (OuterVolumeSpecName: "kube-api-access-t4shk") pod "bef1452a-b8b3-496a-8688-0d0d269a9414" (UID: "bef1452a-b8b3-496a-8688-0d0d269a9414"). InnerVolumeSpecName "kube-api-access-t4shk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:03:10.101275 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.101253    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bef1452a-b8b3-496a-8688-0d0d269a9414" (UID: "bef1452a-b8b3-496a-8688-0d0d269a9414"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:03:10.200342 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.200300    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4shk\" (UniqueName: \"kubernetes.io/projected/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-api-access-t4shk\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:10.200342 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.200333    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bef1452a-b8b3-496a-8688-0d0d269a9414-kserve-provision-location\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:10.200342 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.200344    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-cabundle-cert\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:10.200342 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.200354    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bef1452a-b8b3-496a-8688-0d0d269a9414-proxy-tls\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:10.200626 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.200363    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bef1452a-b8b3-496a-8688-0d0d269a9414-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:10.323149 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.323115    2577 generic.go:356] "Generic (PLEG): container finished" podID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerID="79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8" exitCode=0
Jul 01 19:03:10.323315 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.323173    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"
Jul 01 19:03:10.323315 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.323197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerDied","Data":"79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8"}
Jul 01 19:03:10.323315 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.323256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9" event={"ID":"bef1452a-b8b3-496a-8688-0d0d269a9414","Type":"ContainerDied","Data":"17fb47d9422b6fc11e5c3ff38408fb8ee85e9e147ee1e762375cdad2a040e6ba"}
Jul 01 19:03:10.323315 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.323282    2577 scope.go:117] "RemoveContainer" containerID="018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2"
Jul 01 19:03:10.331464 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.331448    2577 scope.go:117] "RemoveContainer" containerID="79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8"
Jul 01 19:03:10.338771 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.338755    2577 scope.go:117] "RemoveContainer" containerID="c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f"
Jul 01 19:03:10.343800 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.343781    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"]
Jul 01 19:03:10.346852 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.346830    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-hxmd9"]
Jul 01 19:03:10.347259 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.347243    2577 scope.go:117] "RemoveContainer" containerID="018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2"
Jul 01 19:03:10.347503 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:03:10.347485    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2\": container with ID starting with 018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2 not found: ID does not exist" containerID="018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2"
Jul 01 19:03:10.347555 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.347511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2"} err="failed to get container status \"018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2\": rpc error: code = NotFound desc = could not find container \"018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2\": container with ID starting with 018cb7bb919be68fde889f2e80b4b57f36fa42d0d0827c3b7825f66e8d89a2f2 not found: ID does not exist"
Jul 01 19:03:10.347555 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.347529    2577 scope.go:117] "RemoveContainer" containerID="79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8"
Jul 01 19:03:10.347770 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:03:10.347756    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8\": container with ID starting with 79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8 not found: ID does not exist" containerID="79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8"
Jul 01 19:03:10.347850 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.347774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8"} err="failed to get container status \"79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8\": rpc error: code = NotFound desc = could not find container \"79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8\": container with ID starting with 79183c6b4501da0c770f640511da1968c90beac811176d916509f4011082dab8 not found: ID does not exist"
Jul 01 19:03:10.347850 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.347787    2577 scope.go:117] "RemoveContainer" containerID="c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f"
Jul 01 19:03:10.348035 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:03:10.348018    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f\": container with ID starting with c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f not found: ID does not exist" containerID="c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f"
Jul 01 19:03:10.348087 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:10.348044    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f"} err="failed to get container status \"c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f\": rpc error: code = NotFound desc = could not find container \"c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f\": container with ID starting with c48403db2691d655311146865b471c141ebfea7e347811feb681fdb8ac517b2f not found: ID does not exist"
Jul 01 19:03:12.208740 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:12.208712    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" path="/var/lib/kubelet/pods/bef1452a-b8b3-496a-8688-0d0d269a9414/volumes"
Jul 01 19:03:15.867386 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867353    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lf6kp/must-gather-jq8c9"]
Jul 01 19:03:15.867775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container"
Jul 01 19:03:15.867775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867712    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container"
Jul 01 19:03:15.867775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867726    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="storage-initializer"
Jul 01 19:03:15.867775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867731    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="storage-initializer"
Jul 01 19:03:15.867775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867747    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kube-rbac-proxy"
Jul 01 19:03:15.867775 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867752    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kube-rbac-proxy"
Jul 01 19:03:15.867997 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867821    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kube-rbac-proxy"
Jul 01 19:03:15.867997 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.867832    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef1452a-b8b3-496a-8688-0d0d269a9414" containerName="kserve-container"
Jul 01 19:03:15.872406 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.872390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:15.873792 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.873766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:15.874907 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.874889    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lf6kp\"/\"openshift-service-ca.crt\""
Jul 01 19:03:15.875012 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.874988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lf6kp\"/\"kube-root-ca.crt\""
Jul 01 19:03:15.875756 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.875741    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lf6kp\"/\"default-dockercfg-fw25m\""
Jul 01 19:03:15.878537 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.878513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lf6kp/must-gather-jq8c9"]
Jul 01 19:03:15.936308 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.936272    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:15.949443 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.949417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvc2f\" (UniqueName: \"kubernetes.io/projected/f27b98ac-0ea8-4b64-b7a2-def32cd14060-kube-api-access-qvc2f\") pod \"must-gather-jq8c9\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") " pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:15.949543 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:15.949465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f27b98ac-0ea8-4b64-b7a2-def32cd14060-must-gather-output\") pod \"must-gather-jq8c9\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") " pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:16.050574 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.050543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvc2f\" (UniqueName: \"kubernetes.io/projected/f27b98ac-0ea8-4b64-b7a2-def32cd14060-kube-api-access-qvc2f\") pod \"must-gather-jq8c9\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") " pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:16.050725 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.050595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f27b98ac-0ea8-4b64-b7a2-def32cd14060-must-gather-output\") pod \"must-gather-jq8c9\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") " pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:16.050964 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.050946    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f27b98ac-0ea8-4b64-b7a2-def32cd14060-must-gather-output\") pod \"must-gather-jq8c9\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") " pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:16.059722 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.059697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvc2f\" (UniqueName: \"kubernetes.io/projected/f27b98ac-0ea8-4b64-b7a2-def32cd14060-kube-api-access-qvc2f\") pod \"must-gather-jq8c9\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") " pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:16.194676 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.194598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:16.330440 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.330280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lf6kp/must-gather-jq8c9"]
Jul 01 19:03:16.333181 ip-10-0-133-57 kubenswrapper[2577]: W0701 19:03:16.333155    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf27b98ac_0ea8_4b64_b7a2_def32cd14060.slice/crio-ee28dc3582da49f2f09f9a6c0e0105ef2081f4aca377e54eedda2fc27e3f83a2 WatchSource:0}: Error finding container ee28dc3582da49f2f09f9a6c0e0105ef2081f4aca377e54eedda2fc27e3f83a2: Status 404 returned error can't find the container with id ee28dc3582da49f2f09f9a6c0e0105ef2081f4aca377e54eedda2fc27e3f83a2
Jul 01 19:03:16.335244 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.335229    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:03:16.350618 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:16.350587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf6kp/must-gather-jq8c9" event={"ID":"f27b98ac-0ea8-4b64-b7a2-def32cd14060","Type":"ContainerStarted","Data":"ee28dc3582da49f2f09f9a6c0e0105ef2081f4aca377e54eedda2fc27e3f83a2"}
Jul 01 19:03:23.397996 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:23.397955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf6kp/must-gather-jq8c9" event={"ID":"f27b98ac-0ea8-4b64-b7a2-def32cd14060","Type":"ContainerStarted","Data":"9c5fe60a382d854a042858031b02a1c2e321667c0df5e8d6bd4e4dc5181220e3"}
Jul 01 19:03:23.398366 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:23.398006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf6kp/must-gather-jq8c9" event={"ID":"f27b98ac-0ea8-4b64-b7a2-def32cd14060","Type":"ContainerStarted","Data":"aaf6140663ba1b4f93299200f4303e77db1930f69e711b77dffa1fc601395b7a"}
Jul 01 19:03:23.417045 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:23.416999    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lf6kp/must-gather-jq8c9" podStartSLOduration=2.510425365 podStartE2EDuration="8.416982452s" totalImagesPullingTime="5.906557087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:03:23.415217521 +0000 UTC m=+1561.650171800" watchObservedRunningTime="2026-07-01 19:03:23.416982452 +0000 UTC m=+1561.651936717"
Jul 01 19:03:40.476137 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:40.476096    2577 generic.go:356] "Generic (PLEG): container finished" podID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerID="aaf6140663ba1b4f93299200f4303e77db1930f69e711b77dffa1fc601395b7a" exitCode=0
Jul 01 19:03:40.476560 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:40.476172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lf6kp/must-gather-jq8c9" event={"ID":"f27b98ac-0ea8-4b64-b7a2-def32cd14060","Type":"ContainerDied","Data":"aaf6140663ba1b4f93299200f4303e77db1930f69e711b77dffa1fc601395b7a"}
Jul 01 19:03:40.476560 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:40.476476    2577 scope.go:117] "RemoveContainer" containerID="aaf6140663ba1b4f93299200f4303e77db1930f69e711b77dffa1fc601395b7a"
Jul 01 19:03:40.793647 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:40.793569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lf6kp_must-gather-jq8c9_f27b98ac-0ea8-4b64-b7a2-def32cd14060/gather/0.log"
Jul 01 19:03:41.212143 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.212064    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:41.351997 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.351966    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvb65/must-gather-5c9sz"]
Jul 01 19:03:41.354379 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.354361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.357052 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.357032    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvb65\"/\"kube-root-ca.crt\""
Jul 01 19:03:41.357160 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.357054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fvb65\"/\"default-dockercfg-d5w7x\""
Jul 01 19:03:41.357160 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.357108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvb65\"/\"openshift-service-ca.crt\""
Jul 01 19:03:41.364738 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.364717    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvb65/must-gather-5c9sz"]
Jul 01 19:03:41.367706 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.367682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:41.468593 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.468510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:41.469592 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.469571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4729883-d196-4a35-a246-0df024c30888-must-gather-output\") pod \"must-gather-5c9sz\" (UID: \"b4729883-d196-4a35-a246-0df024c30888\") " pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.469653 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.469605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4hp\" (UniqueName: \"kubernetes.io/projected/b4729883-d196-4a35-a246-0df024c30888-kube-api-access-dg4hp\") pod \"must-gather-5c9sz\" (UID: \"b4729883-d196-4a35-a246-0df024c30888\") " pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.570162 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.570126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4729883-d196-4a35-a246-0df024c30888-must-gather-output\") pod \"must-gather-5c9sz\" (UID: \"b4729883-d196-4a35-a246-0df024c30888\") " pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.570162 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.570167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4hp\" (UniqueName: \"kubernetes.io/projected/b4729883-d196-4a35-a246-0df024c30888-kube-api-access-dg4hp\") pod \"must-gather-5c9sz\" (UID: \"b4729883-d196-4a35-a246-0df024c30888\") " pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.570586 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.570464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b4729883-d196-4a35-a246-0df024c30888-must-gather-output\") pod \"must-gather-5c9sz\" (UID: \"b4729883-d196-4a35-a246-0df024c30888\") " pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.578713 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.578688    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4hp\" (UniqueName: \"kubernetes.io/projected/b4729883-d196-4a35-a246-0df024c30888-kube-api-access-dg4hp\") pod \"must-gather-5c9sz\" (UID: \"b4729883-d196-4a35-a246-0df024c30888\") " pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.663990 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.663949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvb65/must-gather-5c9sz"
Jul 01 19:03:41.797949 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:41.797921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvb65/must-gather-5c9sz"]
Jul 01 19:03:41.800225 ip-10-0-133-57 kubenswrapper[2577]: W0701 19:03:41.800199    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4729883_d196_4a35_a246_0df024c30888.slice/crio-60755ef103c999b4ffe9f51522c59a58d1ccb92f05697c79ee42926b110b3dfb WatchSource:0}: Error finding container 60755ef103c999b4ffe9f51522c59a58d1ccb92f05697c79ee42926b110b3dfb: Status 404 returned error can't find the container with id 60755ef103c999b4ffe9f51522c59a58d1ccb92f05697c79ee42926b110b3dfb
Jul 01 19:03:42.485031 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:42.484997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvb65/must-gather-5c9sz" event={"ID":"b4729883-d196-4a35-a246-0df024c30888","Type":"ContainerStarted","Data":"60755ef103c999b4ffe9f51522c59a58d1ccb92f05697c79ee42926b110b3dfb"}
Jul 01 19:03:43.493907 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:43.493855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvb65/must-gather-5c9sz" event={"ID":"b4729883-d196-4a35-a246-0df024c30888","Type":"ContainerStarted","Data":"d746f21046a75007ef7e86357a2e45bd938f6f4c0d926872effbaea8409c2afc"}
Jul 01 19:03:43.493907 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:43.493911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvb65/must-gather-5c9sz" event={"ID":"b4729883-d196-4a35-a246-0df024c30888","Type":"ContainerStarted","Data":"49b965f1b1e94aca905b80d521fd3da97be94494ce0fd12976de8132b48889a5"}
Jul 01 19:03:43.511553 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:43.511472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvb65/must-gather-5c9sz" podStartSLOduration=1.8200881949999999 podStartE2EDuration="2.511452726s" totalImagesPullingTime="691.364531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:03:43.509783061 +0000 UTC m=+1581.744737327" watchObservedRunningTime="2026-07-01 19:03:43.511452726 +0000 UTC m=+1581.746406994"
Jul 01 19:03:44.081099 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:44.081065    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k499p_cba52558-8c44-4d21-94cf-ed28570c6337/global-pull-secret-syncer/0.log"
Jul 01 19:03:44.260129 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:44.260093    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log"
Jul 01 19:03:44.357380 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:44.357304    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-57.ec2.internal_c582d8238a6f3daead6044036f2a5eeb/haproxy/0.log"
Jul 01 19:03:46.190244 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.189529    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-lf6kp/must-gather-jq8c9"]
Jul 01 19:03:46.190244 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.189734    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-lf6kp/must-gather-jq8c9" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="copy" containerID="cri-o://9c5fe60a382d854a042858031b02a1c2e321667c0df5e8d6bd4e4dc5181220e3" gracePeriod=2
Jul 01 19:03:46.193649 ip-10-0-133-57 kubenswrapper[2577]: E0701 19:03:46.193617    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jq8c9\" is forbidden: User \"system:node:ip-10-0-133-57.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lf6kp\": no relationship found between node 'ip-10-0-133-57.ec2.internal' and this object" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:46.193841 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.193754    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-lf6kp/must-gather-jq8c9"]
Jul 01 19:03:46.514521 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.514492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lf6kp_must-gather-jq8c9_f27b98ac-0ea8-4b64-b7a2-def32cd14060/copy/0.log"
Jul 01 19:03:46.515333 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.515296    2577 generic.go:356] "Generic (PLEG): container finished" podID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerID="9c5fe60a382d854a042858031b02a1c2e321667c0df5e8d6bd4e4dc5181220e3" exitCode=143
Jul 01 19:03:46.575143 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.573954    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lf6kp_must-gather-jq8c9_f27b98ac-0ea8-4b64-b7a2-def32cd14060/copy/0.log"
Jul 01 19:03:46.579993 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.579966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:46.617823 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.617593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvc2f\" (UniqueName: \"kubernetes.io/projected/f27b98ac-0ea8-4b64-b7a2-def32cd14060-kube-api-access-qvc2f\") pod \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") "
Jul 01 19:03:46.617823 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.617702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f27b98ac-0ea8-4b64-b7a2-def32cd14060-must-gather-output\") pod \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\" (UID: \"f27b98ac-0ea8-4b64-b7a2-def32cd14060\") "
Jul 01 19:03:46.619210 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.619177    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27b98ac-0ea8-4b64-b7a2-def32cd14060-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f27b98ac-0ea8-4b64-b7a2-def32cd14060" (UID: "f27b98ac-0ea8-4b64-b7a2-def32cd14060"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:03:46.626832 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.625022    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27b98ac-0ea8-4b64-b7a2-def32cd14060-kube-api-access-qvc2f" (OuterVolumeSpecName: "kube-api-access-qvc2f") pod "f27b98ac-0ea8-4b64-b7a2-def32cd14060" (UID: "f27b98ac-0ea8-4b64-b7a2-def32cd14060"). InnerVolumeSpecName "kube-api-access-qvc2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:03:46.718996 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.718935    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f27b98ac-0ea8-4b64-b7a2-def32cd14060-must-gather-output\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:46.718996 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.718970    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvc2f\" (UniqueName: \"kubernetes.io/projected/f27b98ac-0ea8-4b64-b7a2-def32cd14060-kube-api-access-qvc2f\") on node \"ip-10-0-133-57.ec2.internal\" DevicePath \"\""
Jul 01 19:03:46.991720 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:46.991675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:47.073338 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:47.073295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29kff_9106ec84-55be-4d4e-be47-97aca4c61dc4/konnectivity-agent/0.log" limit=1024
Jul 01 19:03:47.523825 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:47.523242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lf6kp_must-gather-jq8c9_f27b98ac-0ea8-4b64-b7a2-def32cd14060/copy/0.log"
Jul 01 19:03:47.524320 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:47.523945    2577 scope.go:117] "RemoveContainer" containerID="9c5fe60a382d854a042858031b02a1c2e321667c0df5e8d6bd4e4dc5181220e3"
Jul 01 19:03:47.524320 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:47.524131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lf6kp/must-gather-jq8c9"
Jul 01 19:03:47.543174 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:47.539323    2577 scope.go:117] "RemoveContainer" containerID="aaf6140663ba1b4f93299200f4303e77db1930f69e711b77dffa1fc601395b7a"
Jul 01 19:03:48.212115 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:48.212073    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" path="/var/lib/kubelet/pods/f27b98ac-0ea8-4b64-b7a2-def32cd14060/volumes"
Jul 01 19:03:48.842368 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:48.842280    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-5dvqm_728e3048-f973-425d-9c5f-27a99f6295b5/cluster-monitoring-operator/0.log"
Jul 01 19:03:49.016939 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.016913    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-qbvsz_ab61fa60-f027-44bf-a598-9ae1532455b3/monitoring-plugin/0.log"
Jul 01 19:03:49.145328 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.145230    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc6zk_ca8c047d-edd8-453e-ba5f-e2e62c926a23/node-exporter/0.log"
Jul 01 19:03:49.177462 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.177437    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc6zk_ca8c047d-edd8-453e-ba5f-e2e62c926a23/kube-rbac-proxy/0.log"
Jul 01 19:03:49.211678 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.211647    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc6zk_ca8c047d-edd8-453e-ba5f-e2e62c926a23/init-textfile/0.log"
Jul 01 19:03:49.413752 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.413680    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-l74kl_5ae8c8ef-7358-4be8-8a97-283cda0d6eb7/kube-rbac-proxy-main/0.log"
Jul 01 19:03:49.441907 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.441879    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-l74kl_5ae8c8ef-7358-4be8-8a97-283cda0d6eb7/kube-rbac-proxy-self/0.log"
Jul 01 19:03:49.472568 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.472535    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-l74kl_5ae8c8ef-7358-4be8-8a97-283cda0d6eb7/openshift-state-metrics/0.log"
Jul 01 19:03:49.516776 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.516746    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/prometheus/0.log"
Jul 01 19:03:49.547247 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.547215    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/config-reloader/0.log"
Jul 01 19:03:49.584209 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.584189    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/thanos-sidecar/0.log"
Jul 01 19:03:49.612536 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.612512    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/kube-rbac-proxy-web/0.log"
Jul 01 19:03:49.659630 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.659601    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/kube-rbac-proxy/0.log"
Jul 01 19:03:49.685351 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.685232    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/kube-rbac-proxy-thanos/0.log"
Jul 01 19:03:49.711089 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.711060    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_33e70a4a-bf63-4524-9040-78c6b60b2d1a/init-config-reloader/0.log"
Jul 01 19:03:49.750541 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.750507    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-bk6ch_20a0f5be-9501-40d5-af2e-e298213cf9e6/prometheus-operator/0.log"
Jul 01 19:03:49.774556 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.774509    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-bk6ch_20a0f5be-9501-40d5-af2e-e298213cf9e6/kube-rbac-proxy/0.log"
Jul 01 19:03:49.804452 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.804420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7ddc8784d4-k6zwt_b3fa3bbc-7729-46bc-b759-0b863aed1c22/prometheus-operator-admission-webhook/0.log"
Jul 01 19:03:49.956402 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.956321    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645f96b865-npct9_cc539bb8-2af5-4c86-83fc-d5eb5046950a/thanos-query/0.log"
Jul 01 19:03:49.994033 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:49.993958    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645f96b865-npct9_cc539bb8-2af5-4c86-83fc-d5eb5046950a/kube-rbac-proxy-web/0.log"
Jul 01 19:03:50.025120 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:50.025095    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645f96b865-npct9_cc539bb8-2af5-4c86-83fc-d5eb5046950a/kube-rbac-proxy/0.log"
Jul 01 19:03:50.052908 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:50.052882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645f96b865-npct9_cc539bb8-2af5-4c86-83fc-d5eb5046950a/prom-label-proxy/0.log"
Jul 01 19:03:50.090161 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:50.090135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645f96b865-npct9_cc539bb8-2af5-4c86-83fc-d5eb5046950a/kube-rbac-proxy-rules/0.log"
Jul 01 19:03:50.127418 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:50.127391    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645f96b865-npct9_cc539bb8-2af5-4c86-83fc-d5eb5046950a/kube-rbac-proxy-metrics/0.log"
Jul 01 19:03:51.417209 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:51.417126    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-cw2s5_a97dee06-2aa7-476b-9ca1-03755b87c713/networking-console-plugin/0.log"
Jul 01 19:03:51.938271 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:51.938239    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/2.log"
Jul 01 19:03:51.943920 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:51.943891    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-jnztt_101b5c81-fe81-4627-9485-a46d1c8844b1/console-operator/3.log"
Jul 01 19:03:52.432076 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.432051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-qkt6x_f2fe2315-f9aa-426d-9adf-93ff2f0bd73f/download-server/0.log"
Jul 01 19:03:52.486531 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.486497    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"]
Jul 01 19:03:52.487266 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.487242    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="gather"
Jul 01 19:03:52.487420 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.487394    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="gather"
Jul 01 19:03:52.487551 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.487445    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="copy"
Jul 01 19:03:52.487551 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.487454    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="copy"
Jul 01 19:03:52.487551 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.487546    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="copy"
Jul 01 19:03:52.487725 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.487557    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27b98ac-0ea8-4b64-b7a2-def32cd14060" containerName="gather"
Jul 01 19:03:52.493051 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.493020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.499092 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.499066    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"]
Jul 01 19:03:52.579130 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.579092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-proc\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.579336 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.579143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-lib-modules\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.579336 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.579206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-sys\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.579336 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.579267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tk6\" (UniqueName: \"kubernetes.io/projected/851d94e8-036f-40eb-93a8-578740f234a6-kube-api-access-n6tk6\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.579336 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.579284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-podres\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.680329 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.680287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-sys\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.680627 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.680605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tk6\" (UniqueName: \"kubernetes.io/projected/851d94e8-036f-40eb-93a8-578740f234a6-kube-api-access-n6tk6\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.680761 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.680745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-podres\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.680896 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.680880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-proc\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.681515 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.681009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-lib-modules\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.681515 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.681034    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-proc\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.681515 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.680490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-sys\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.681515 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.681222    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-lib-modules\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.681515 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.681496    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/851d94e8-036f-40eb-93a8-578740f234a6-podres\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.690918 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.690862    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tk6\" (UniqueName: \"kubernetes.io/projected/851d94e8-036f-40eb-93a8-578740f234a6-kube-api-access-n6tk6\") pod \"perf-node-gather-daemonset-rnwb5\" (UID: \"851d94e8-036f-40eb-93a8-578740f234a6\") " pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.810926 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.810892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:52.991832 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:52.991615    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"]
Jul 01 19:03:52.994748 ip-10-0-133-57 kubenswrapper[2577]: W0701 19:03:52.994720    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod851d94e8_036f_40eb_93a8_578740f234a6.slice/crio-9cf12723a27d46c4dcfc6a5234928353cbcc11a8ad29770f2d90fe68e3e488f6 WatchSource:0}: Error finding container 9cf12723a27d46c4dcfc6a5234928353cbcc11a8ad29770f2d90fe68e3e488f6: Status 404 returned error can't find the container with id 9cf12723a27d46c4dcfc6a5234928353cbcc11a8ad29770f2d90fe68e3e488f6
Jul 01 19:03:53.005974 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.005944    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-cmldk_a836d8e1-cc91-4cf3-ada8-b1b5955b63dc/volume-data-source-validator/0.log"
Jul 01 19:03:53.563845 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.563792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5" event={"ID":"851d94e8-036f-40eb-93a8-578740f234a6","Type":"ContainerStarted","Data":"2fd9defb55b34f113e2cffed2ebe0c70db6da3ae858b0e415adaa8c9266151bf"}
Jul 01 19:03:53.564323 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.563873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5" event={"ID":"851d94e8-036f-40eb-93a8-578740f234a6","Type":"ContainerStarted","Data":"9cf12723a27d46c4dcfc6a5234928353cbcc11a8ad29770f2d90fe68e3e488f6"}
Jul 01 19:03:53.564323 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.563939    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:03:53.580395 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.580332    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5" podStartSLOduration=1.5803138909999999 podStartE2EDuration="1.580313891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:03:53.579186704 +0000 UTC m=+1591.814140984" watchObservedRunningTime="2026-07-01 19:03:53.580313891 +0000 UTC m=+1591.815268157"
Jul 01 19:03:53.924433 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.924365    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9s8cn_6b9359a6-a671-416d-89cd-fc18afd1fdc8/dns/0.log"
Jul 01 19:03:53.950752 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:53.950716    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9s8cn_6b9359a6-a671-416d-89cd-fc18afd1fdc8/kube-rbac-proxy/0.log"
Jul 01 19:03:54.123765 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:54.123739    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v6j5f_1fa76cfd-880b-4b2a-a285-e9fc3c4a9a7d/dns-node-resolver/0.log"
Jul 01 19:03:54.667880 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:54.667849    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k5qgm_9cb73641-462d-4970-98c5-d491835f923f/node-ca/0.log"
Jul 01 19:03:55.978624 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:55.978594    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-skvhh_6d13004e-5a72-4440-8c2d-df5ea9e56024/serve-healthcheck-canary/0.log"
Jul 01 19:03:56.565947 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:56.565919    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gpcfz_375d8359-feea-4c7a-b45e-169f4badc3e4/kube-rbac-proxy/0.log"
Jul 01 19:03:56.590739 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:56.590706    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gpcfz_375d8359-feea-4c7a-b45e-169f4badc3e4/exporter/0.log"
Jul 01 19:03:56.615229 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:56.615200    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gpcfz_375d8359-feea-4c7a-b45e-169f4badc3e4/extractor/0.log"
Jul 01 19:03:58.860431 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:58.860402    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4n865_0e819c7b-bfb6-4f40-9a29-3d06239a26ea/s3-init/0.log"
Jul 01 19:03:58.907983 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:58.907941    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4n865_0e819c7b-bfb6-4f40-9a29-3d06239a26ea/download-hf-model/0.log"
Jul 01 19:03:58.936130 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:58.936098    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-wtcxn_4c2bed35-c921-41dd-a235-38a4d6703c66/s3-tls-init-custom/0.log"
Jul 01 19:03:58.960133 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:58.960098    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-wtcxn_4c2bed35-c921-41dd-a235-38a4d6703c66/download-hf-model/0.log"
Jul 01 19:03:58.990885 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:58.990860    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-d66b8_3d47e275-d2e7-470a-8a65-de73969ee068/s3-tls-init-serving/0.log"
Jul 01 19:03:59.013095 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:59.013068    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-d66b8_3d47e275-d2e7-470a-8a65-de73969ee068/download-hf-model/0.log"
Jul 01 19:03:59.068207 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:59.068183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-qdtc5_41b69c0c-500c-4997-b3fb-46461ab0e6f2/seaweedfs-tls-custom/0.log"
Jul 01 19:03:59.095776 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:59.095729    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-hpwrt_73e0e2ff-145e-49b5-bf44-0a3c1f4ef8f0/seaweedfs-tls-serving/0.log"
Jul 01 19:03:59.582849 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:03:59.582817    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fvb65/perf-node-gather-daemonset-rnwb5"
Jul 01 19:04:04.967790 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:04.967712    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/kube-multus-additional-cni-plugins/0.log"
Jul 01 19:04:04.992631 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:04.992598    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/egress-router-binary-copy/0.log"
Jul 01 19:04:05.016578 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.016555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/cni-plugins/0.log"
Jul 01 19:04:05.039532 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.039503    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/bond-cni-plugin/0.log"
Jul 01 19:04:05.065108 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.065078    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/routeoverride-cni/0.log"
Jul 01 19:04:05.090271 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.090243    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/whereabouts-cni-bincopy/0.log"
Jul 01 19:04:05.150730 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.150698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-smlvd_3ebfb66f-330b-495f-bd2b-4c25c6b2f043/whereabouts-cni/0.log"
Jul 01 19:04:05.443721 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.443693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wqlzd_c9ce6483-0300-475e-855c-e5237856a12f/kube-multus/0.log"
Jul 01 19:04:05.584190 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.584159    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xkqt7_9f6cb189-4786-4e41-bd91-0888314a0b65/network-metrics-daemon/0.log"
Jul 01 19:04:05.608426 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:05.608399    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xkqt7_9f6cb189-4786-4e41-bd91-0888314a0b65/kube-rbac-proxy/0.log"
Jul 01 19:04:06.607687 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.607660    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-controller/0.log"
Jul 01 19:04:06.626438 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.626412    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/1.log"
Jul 01 19:04:06.626603 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.626540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovn-acl-logging/0.log"
Jul 01 19:04:06.647139 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.647113    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/kube-rbac-proxy-node/0.log"
Jul 01 19:04:06.672238 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.672205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 19:04:06.696546 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.696516    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/northd/0.log"
Jul 01 19:04:06.719350 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.719328    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/nbdb/0.log"
Jul 01 19:04:06.743039 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.743009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/sbdb/0.log"
Jul 01 19:04:06.786709 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:06.786679    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4bhn_0aa73a80-6429-4a23-8445-6b1110dddd5b/ovnkube-controller/0.log"
Jul 01 19:04:08.044133 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:08.044102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-c69kt_11574496-4380-4df8-a38b-5e732c84d264/network-check-target-container/0.log"
Jul 01 19:04:09.078397 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:09.078372    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9wmgw_28885980-38ee-46ba-9944-e7ce6739496e/iptables-alerter/0.log"
Jul 01 19:04:09.851069 ip-10-0-133-57 kubenswrapper[2577]: I0701 19:04:09.851038    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-w692r_8aca6411-d820-4c64-bec2-2ef05551c627/tuned/0.log"