Jul 03 20:35:09.802125 ip-10-0-129-236 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 03 20:35:09.802135 ip-10-0-129-236 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 03 20:35:09.802142 ip-10-0-129-236 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 03 20:35:09.802429 ip-10-0-129-236 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 03 20:35:19.440700 ip-10-0-129-236 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 03 20:35:19.440713 ip-10-0-129-236 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c3c8af331dd04574a7b305a3605e3ab0 --
Jul 03 20:37:07.807220 ip-10-0-129-236 systemd[1]: Starting Kubernetes Kubelet...
Jul 03 20:37:08.186462 ip-10-0-129-236 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 03 20:37:08.186462 ip-10-0-129-236 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 03 20:37:08.186462 ip-10-0-129-236 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 03 20:37:08.186462 ip-10-0-129-236 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 03 20:37:08.186462 ip-10-0-129-236 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 03 20:37:08.187273 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.186674    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192049    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192062    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192066    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192069    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192072    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192075    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192078    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192081    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192084    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192087    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192090    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192092    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192095    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192098    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192101    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192104    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192106    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192110    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192112    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 03 20:37:08.302121 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192120    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 03 20:37:08.261094 ip-10-0-129-236 systemd[1]: Started Kubernetes Kubelet.
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192123    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192126    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192128    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192131    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192133    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192136    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192138    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192141    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192143    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192146    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192149    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192151    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192154    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192156    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192159    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192161    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192164    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192166    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192169    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 03 20:37:08.302907 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192171    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192174    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192177    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192179    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192182    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192185    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192187    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192190    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192193    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192195    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192198    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192201    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192203    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192206    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192208    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192210    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192213    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192216    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192218    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192220    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 03 20:37:08.315802 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192223    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192225    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192228    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192231    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192236    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192240    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192244    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192247    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192250    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192252    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192255    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192258    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192261    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192267    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192270    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192273    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192275    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192278    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192280    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 03 20:37:08.317087 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192283    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192286    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192289    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192291    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192294    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192297    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192299    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192302    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192305    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192307    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192311    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192314    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192317    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192320    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192323    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192326    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192329    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192332    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192335    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 03 20:37:08.317950 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192337    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.192340    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194259    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194269    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194272    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194276    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194279    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194283    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194286    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194289    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194292    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194294    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194297    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194301    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194303    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194306    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194308    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194311    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194313    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 03 20:37:08.318799 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194316    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194320    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194322    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194325    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194328    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194330    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194333    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194335    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194338    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194341    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194343    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194346    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194349    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194351    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194354    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194358    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194362    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194365    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194367    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194370    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 03 20:37:08.319670 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194374    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194377    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194379    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194382    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194385    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194389    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194392    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194396    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194398    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194401    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194403    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194406    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194408    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194411    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194414    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194416    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194421    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194424    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194427    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 03 20:37:08.320959 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194430    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194433    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194436    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194438    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194441    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194444    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194446    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194449    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194452    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194455    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194457    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194459    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194462    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194464    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194467    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194470    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194472    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194474    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194477    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194480    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 03 20:37:08.321839 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194482    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194485    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194487    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194490    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194492    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194495    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194498    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194501    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194503    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194506    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194509    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194512    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194514    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194518    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194521    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194524    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194526    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194528    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194531    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194534    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 03 20:37:08.323567 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194536    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194538    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.194541    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195017    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195023    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195027    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195030    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195033    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195037    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195041    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195044    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195047    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195050    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195054    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195056    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195060    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195063    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195066    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195068    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 03 20:37:08.324869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195071    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195074    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195077    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195079    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195082    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195084    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195087    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195090    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195092    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195095    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195098    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195101    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195103    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195107    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195110    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195114    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195117    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195120    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195123    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 03 20:37:08.430038 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195126    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195129    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195131    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195134    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195137    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195139    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195142    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195146    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195156    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195159    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195162    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195164    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195166    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195169    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195172    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195174    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195176    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195179    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195181    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195184    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 03 20:37:08.430824 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195186    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195189    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195192    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195195    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195197    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195200    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195202    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195205    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195207    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195210    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195212    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195215    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195217    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195219    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195222    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195224    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195228    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195231    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195234    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195237    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 03 20:37:08.431696 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195241    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195244    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195246    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195249    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195252    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195255    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195257    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195260    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195263    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195265    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195268    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195270    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195273    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195276    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195278    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195281    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195283    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195286    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195288    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195291    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 03 20:37:08.432561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195294    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195296    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195299    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.195301    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195364    2553 flags.go:64] FLAG: --address="0.0.0.0"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195375    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195382    2553 flags.go:64] FLAG: --anonymous-auth="true"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195387    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195390    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195394    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195398    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195402    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195406    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195409    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195413    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195416    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195420    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195423    2553 flags.go:64] FLAG: --cgroup-root=""
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195426    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195428    2553 flags.go:64] FLAG: --client-ca-file=""
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195431    2553 flags.go:64] FLAG: --cloud-provider="external"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195436    2553 flags.go:64] FLAG: --cluster-dns="[]"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195439    2553 flags.go:64] FLAG: --cluster-domain=""
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195442    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 03 20:37:08.433429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195445    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195448    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195451    2553 flags.go:64] FLAG: --container-log-max-files="5"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195455    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195458    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195461    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195464    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195467    2553 flags.go:64] FLAG: --contention-profiling="false"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195470    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195473    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195476    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195479    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195483    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195486    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195489    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195491    2553 flags.go:64] FLAG: --enable-load-reader="false"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195494    2553 flags.go:64] FLAG: --enable-server="true"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195497    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195502    2553 flags.go:64] FLAG: --event-burst="100"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195504    2553 flags.go:64] FLAG: --event-qps="50"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195508    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195511    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195514    2553 flags.go:64] FLAG: --eviction-hard=""
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195518    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 03 20:37:08.434715 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195521    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195524    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195527    2553 flags.go:64] FLAG: --eviction-soft=""
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195530    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195533    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195538    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195541    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195544    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195547    2553 flags.go:64] FLAG: --fail-swap-on="true"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195549    2553 flags.go:64] FLAG: --feature-gates=""
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195558    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195561    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195564    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195567    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195570    2553 flags.go:64] FLAG: --healthz-port="10248"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195573    2553 flags.go:64] FLAG: --help="false"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195576    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-129-236.ec2.internal"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195579    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195581    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195584    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195588    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195591    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195594    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 03 20:37:08.436487 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195596    2553 flags.go:64] FLAG: --image-service-endpoint=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195599    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195602    2553 flags.go:64] FLAG: --kube-api-burst="100"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195605    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195608    2553 flags.go:64] FLAG: --kube-api-qps="50"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195614    2553 flags.go:64] FLAG: --kube-reserved=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195619    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195623    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195626    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195629    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195632    2553 flags.go:64] FLAG: --lock-file=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195635    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195638    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195641    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195648    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195652    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195657    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195661    2553 flags.go:64] FLAG: --logging-format="text"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195664    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195668    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195671    2553 flags.go:64] FLAG: --manifest-url=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195674    2553 flags.go:64] FLAG: --manifest-url-header=""
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195678    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195681    2553 flags.go:64] FLAG: --max-open-files="1000000"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195685    2553 flags.go:64] FLAG: --max-pods="110"
Jul 03 20:37:08.437777 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195688    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195691    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195694    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195697    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195700    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195702    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195705    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195712    2553 flags.go:64] FLAG: --node-status-max-images="50"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195715    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195718    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195721    2553 flags.go:64] FLAG: --pod-cidr=""
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195724    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195731    2553 flags.go:64] FLAG: --pod-manifest-path=""
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195733    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195737    2553 flags.go:64] FLAG: --pods-per-core="0"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195739    2553 flags.go:64] FLAG: --port="10250"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195742    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195746    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c941868fade781cb"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195749    2553 flags.go:64] FLAG: --qos-reserved=""
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195752    2553 flags.go:64] FLAG: --read-only-port="10255"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195755    2553 flags.go:64] FLAG: --register-node="true"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195758    2553 flags.go:64] FLAG: --register-with-taints=""
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195763    2553 flags.go:64] FLAG: --registry-burst="10"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195766    2553 flags.go:64] FLAG: --registry-qps="5"
Jul 03 20:37:08.439533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195769    2553 flags.go:64] FLAG: --reserved-cpus=""
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195771    2553 flags.go:64] FLAG: --reserved-memory=""
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195775    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195778    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195781    2553 flags.go:64] FLAG: --rotate-certificates="false"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195783    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195786    2553 flags.go:64] FLAG: --runonce="false"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195789    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195792    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195795    2553 flags.go:64] FLAG: --seccomp-default="false"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195798    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195801    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195804    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195807    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195810    2553 flags.go:64] FLAG: --storage-driver-password="root"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195812    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195815    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195818    2553 flags.go:64] FLAG: --storage-driver-user="root"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195821    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195824    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195828    2553 flags.go:64] FLAG: --system-cgroups=""
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195830    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195835    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195838    2553 flags.go:64] FLAG: --tls-cert-file=""
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195841    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 03 20:37:08.440819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195845    2553 flags.go:64] FLAG: --tls-min-version=""
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195848    2553 flags.go:64] FLAG: --tls-private-key-file=""
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195851    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195853    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195856    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195860    2553 flags.go:64] FLAG: --v="2"
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195864    2553 flags.go:64] FLAG: --version="false"
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195868    2553 flags.go:64] FLAG: --vmodule=""
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195871    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.195874    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196013    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196018    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196022    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196025    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196029    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196031    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196034    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196036    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196039    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196041    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196043    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 03 20:37:08.442122 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196046    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196048    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196051    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196054    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196056    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196058    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196062    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196065    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196068    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196070    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196072    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196075    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196078    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196081    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196084    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196086    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196091    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196093    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196097    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196101    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 03 20:37:08.446525 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196104    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196106    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196109    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196111    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196114    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196118    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196121    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196124    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196127    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196129    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196132    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196134    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196137    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196139    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196142    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196144    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196147    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196149    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196153    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196156    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 03 20:37:08.447391 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196158    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196161    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196163    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196166    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196168    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196171    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196174    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196176    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196180    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196183    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196185    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196188    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196190    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196193    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196196    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196198    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196200    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196203    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196206    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196208    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 03 20:37:08.449159 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196212    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196216    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196218    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196221    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196224    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196227    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196229    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196232    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196234    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196237    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196241    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196243    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196246    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196248    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196251    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196254    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196256    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196259    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196262    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 03 20:37:08.450016 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196265    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196269    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196271    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196274    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196276    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196279    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196282    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196284    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.196287    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.196292    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.203759    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.203773    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.203791    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.203797    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203863    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 03 20:37:08.451322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203868    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203872    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203875    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203878    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203881    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203884    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203887    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203904    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203907    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203910    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203912    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203915    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203918    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203920    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203923    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203925    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203928    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203939    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203941    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 03 20:37:08.452190 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203944    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203947    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203949    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203953    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203955    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203958    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203960    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203963    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203966    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203969    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203973    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203977    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203980    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203983    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203985    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203988    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203991    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203993    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203996    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 03 20:37:08.564105 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.203999    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204001    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204004    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204010    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204013    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204016    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204018    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204021    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204023    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204026    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204029    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204031    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204039    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204041    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204044    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204047    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204050    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204052    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204055    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204057    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 03 20:37:08.564944 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204060    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204063    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204068    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204072    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204076    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204079    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204082    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204084    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204087    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204089    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204092    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204094    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204097    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204099    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204102    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204105    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204108    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204111    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204113    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204116    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 03 20:37:08.567115 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204118    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204121    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204124    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204127    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204130    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204137    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204140    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204143    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204145    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204150    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204152    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204155    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204159    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204162    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204166    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204169    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204172    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204174    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204177    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 03 20:37:08.567993 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204180    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.204185    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204357    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204362    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204365    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204368    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204371    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204374    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204377    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204380    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204382    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204385    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204388    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204390    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204393    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 03 20:37:08.568868 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204395    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204398    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204401    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204403    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204411    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204413    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204416    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204418    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204422    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204425    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204427    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204430    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204434    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204436    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204439    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204442    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204444    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204447    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204449    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204452    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 03 20:37:08.576252 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204454    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204457    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204459    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204462    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204464    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204467    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204469    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204472    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204474    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204477    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204479    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204482    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204484    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204487    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204490    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204494    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204497    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204511    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204514    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204518    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 03 20:37:08.577110 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204525    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204535    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204539    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204542    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204546    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204549    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204552    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204554    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204557    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204560    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204563    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204566    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204569    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204571    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204574    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204577    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204579    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204581    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204584    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204587    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 03 20:37:08.578009 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204589    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204592    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204594    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204598    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204602    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204605    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204607    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204610    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204613    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204615    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204625    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204628    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204632    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204635    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204637    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204640    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204644    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204646    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204649    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 03 20:37:08.580606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204651    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204654    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204656    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204659    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204661    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204664    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:08.204666    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.204671    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.204772    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.206805    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.208298    2553 server.go:1019] "Starting client certificate rotation"
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.208410    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.208443    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 03 20:37:08.581474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.210711    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.214997    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.230171    2553 log.go:25] "Validated CRI v1 runtime API"
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.232160    2553 log.go:25] "Validated CRI v1 image API"
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.233042    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.235418    2553 fs.go:135] Filesystem UUIDs: map[0f8e82d3-f3c4-444c-aedc-a912ed6aadf1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 82cff5e4-1dd3-4c37-912e-4b96b5c5e4db:/dev/nvme0n1p3]
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.235432    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 03 20:37:08.581904 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.235964    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.240592    2553 manager.go:216] Machine: {Timestamp:2026-07-03 20:37:08.239286925 +0000 UTC m=+0.320028325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101779 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df0d7bf9292ce10f2d14f104ee7ed SystemUUID:ec2df0d7-bf92-92ce-10f2-d14f104ee7ed BootID:c3c8af33-1dd0-4574-a7b3-05a3605e3ab0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:98:4b:eb:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:98:4b:eb:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:a1:05:16:42:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.240700    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.240808    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.243568    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.243590    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.244203    2553 topology_manager.go:138] "Creating topology manager with none policy"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.244213    2553 container_manager_linux.go:309] "Creating device plugin manager"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.244220    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.244762    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.245992    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249492    2553 state_mem.go:36] "Initialized new in-memory state store"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249606    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249632    2553 kubelet.go:486] "Attempting to sync node with API server"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249642    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249655    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249680    2553 kubelet.go:398] "Adding apiserver pod source"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.249688    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 03 20:37:08.584969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.250030    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.250046    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.252064    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.254051    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.254078    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255168    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255190    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255196    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255201    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255210    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255215    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255223    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255228    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255234    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255240    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255252    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255258    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255962    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.255970    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.260044    2553 server.go:1268] "Started kubelet"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.260122    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.260140    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.260213    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.261547    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.261717    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.261921    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.262048    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.265772    2553 server.go:310] "Adding debug handlers to kubelet server"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.266364    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.268540    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-236.ec2.internal.18bee2086a84ce07  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-236.ec2.internal,UID:ip-10-0-129-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-236.ec2.internal,},FirstTimestamp:2026-07-03 20:37:08.259999239 +0000 UTC m=+0.340740637,LastTimestamp:2026-07-03 20:37:08.259999239 +0000 UTC m=+0.340740637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-236.ec2.internal,}"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.270831    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.270860    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.270935    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271544    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271607    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271621    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271802    2553 reconstruct.go:97] "Volume reconstruction finished"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271811    2553 reconciler.go:29] "Reconciler: start to sync state"
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271821    2553 factory.go:55] Registering systemd factory
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.271837    2553 factory.go:223] Registration of the systemd container factory successfully
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.272107    2553 factory.go:153] Registering CRI-O factory
Jul 03 20:37:08.587142 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.272119    2553 factory.go:223] Registration of the crio container factory successfully
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.272169    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.272192    2553 factory.go:103] Registering Raw factory
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.272205    2553 manager.go:1196] Started watching for new ooms in manager
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.272511    2553 manager.go:349] Starting recovery of all containers
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.283756    2553 manager.go:354] Recovery completed
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.288014    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.289916    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.289962    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.289985    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.290602    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.290609    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.290614    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.290634    2553 state_mem.go:36] "Initialized new in-memory state store"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.291432    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.291464    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.291505    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.292929    2553 policy_none.go:49] "None policy: Start"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.292943    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.292954    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.295158    2553 policy_none.go:47] "Start"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.297307    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v4bk4"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.298939    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.299003    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.298879    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-236.ec2.internal.18bee2086c4db19e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-236.ec2.internal,UID:ip-10-0-129-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-236.ec2.internal,},FirstTimestamp:2026-07-03 20:37:08.289941918 +0000 UTC m=+0.370683323,LastTimestamp:2026-07-03 20:37:08.289941918 +0000 UTC m=+0.370683323,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-236.ec2.internal,}"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.305644    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v4bk4"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.371668    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.472021    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.588437 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.572538    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.693502 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.672801    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.693502 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.680185    2553 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-236.ec2.internal\" not found" node="ip-10-0-129-236.ec2.internal"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.706576    2553 manager.go:337] "Starting Device Plugin manager"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.706608    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.706618    2553 server.go:87] "Starting device plugin registration server"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.706871    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.707029    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.707135    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.707223    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.707229    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.707764    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.707803    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.808962    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.810700    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.810733    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.810745    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:08.816653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.810772    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-236.ec2.internal"
Jul 03 20:37:08.820439 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.820189    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-236.ec2.internal"
Jul 03 20:37:08.820439 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.820213    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-236.ec2.internal\": node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.836776    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.862408    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.863789    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.863827    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.863848    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.863890    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.866642    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 03 20:37:08.945617 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:08.937095    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.965347    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"]
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.965431    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.966472    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.966498    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.966508    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.968823    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.969226    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.969250    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.969963    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.969982    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.969991    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.970265    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.970280    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.970289    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.972366    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.972388    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.973160    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientMemory"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.973181    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasNoDiskPressure"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.973191    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeHasSufficientPID"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.979754    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fae37a88080d71f4e174849eeb91a89e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" (UID: \"fae37a88080d71f4e174849eeb91a89e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.979775    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fae37a88080d71f4e174849eeb91a89e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" (UID: \"fae37a88080d71f4e174849eeb91a89e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:08.979793    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d17de08da950bb8f59f5515256e0343-config\") pod \"kube-apiserver-proxy-ip-10-0-129-236.ec2.internal\" (UID: \"4d17de08da950bb8f59f5515256e0343\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.085720 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:09.037662    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:09.215399 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:09.138444    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:09.238755    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-236.ec2.internal\" not found"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286095    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286148    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fae37a88080d71f4e174849eeb91a89e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" (UID: \"fae37a88080d71f4e174849eeb91a89e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286182    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fae37a88080d71f4e174849eeb91a89e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" (UID: \"fae37a88080d71f4e174849eeb91a89e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d17de08da950bb8f59f5515256e0343-config\") pod \"kube-apiserver-proxy-ip-10-0-129-236.ec2.internal\" (UID: \"4d17de08da950bb8f59f5515256e0343\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286269    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fae37a88080d71f4e174849eeb91a89e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" (UID: \"fae37a88080d71f4e174849eeb91a89e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286278    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d17de08da950bb8f59f5515256e0343-config\") pod \"kube-apiserver-proxy-ip-10-0-129-236.ec2.internal\" (UID: \"4d17de08da950bb8f59f5515256e0343\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286285    2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.286359    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fae37a88080d71f4e174849eeb91a89e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" (UID: \"fae37a88080d71f4e174849eeb91a89e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.299912    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.308543    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-02 20:32:09 +0000 UTC" deadline="2028-02-04 23:30:03.324989496 +0000 UTC"
Jul 03 20:37:09.347785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.308566    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13946h52m54.016426737s"
Jul 03 20:37:09.476289 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.372034    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.476289 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.388535    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 03 20:37:09.476289 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.390539    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.476289 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.426437    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 03 20:37:09.661987 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.661965    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.666779 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.666763    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.679310 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.679290    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 03 20:37:09.689440 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.689423    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 03 20:37:09.693311 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.693294    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 03 20:37:09.693367 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:09.693335    2553 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.695954 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.695939    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.705372 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.705353    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 03 20:37:09.705447 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:09.705401    2553 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:09.708424 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:09.708406    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal"
Jul 03 20:37:10.271944 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.271919    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 03 20:37:10.283894 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.283871    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 03 20:37:10.286231 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.286213    2553 apiserver.go:52] "Watching apiserver"
Jul 03 20:37:10.295457 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.295436    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 03 20:37:10.296382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.296359    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xwkdt","kube-system/konnectivity-agent-4bgfb","kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal","openshift-dns/node-resolver-zpjjr","openshift-image-registry/node-ca-rcmn4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal","openshift-multus/multus-9p4vf","openshift-multus/multus-additional-cni-plugins-jsnzb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7","openshift-cluster-node-tuning-operator/tuned-2tgxt","openshift-multus/network-metrics-daemon-ts9j8","openshift-network-diagnostics/network-check-target-qf5hm","openshift-network-operator/iptables-alerter-l7rqd","openshift-ovn-kubernetes/ovnkube-node-l4mn2"]
Jul 03 20:37:10.299204 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.299175    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.301419 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.301394    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.302107 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.302085    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 03 20:37:10.302255 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.302236    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 03 20:37:10.302731 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.302713    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.302896 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.302878    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mw6zl\""
Jul 03 20:37:10.302940 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.302892    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.303880 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.303863    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.304393 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.304372    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.304486 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.304429    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.304486 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.304436    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 03 20:37:10.304594 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.304545    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rplcp\""
Jul 03 20:37:10.306278 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.306261    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.306442 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.306427    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pj9df\""
Jul 03 20:37:10.306507 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.306485    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 03 20:37:10.307291 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.307271    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-prh5b"
Jul 03 20:37:10.307291 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.307273    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 03 20:37:10.307381 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.307295    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.307381 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.307295    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 03 20:37:10.308008 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.307989    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.308131 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.308028    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 03 20:37:10.308644 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.308629    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.309173 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.309085    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.309173 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.309117    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89j8v\""
Jul 03 20:37:10.309611 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.309595    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.311980 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.311324    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.315590 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.315567    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.316116 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.316088    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-prh5b"
Jul 03 20:37:10.316932 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.316903    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae37a88080d71f4e174849eeb91a89e.slice/crio-b1b48bd14c7b60959b504fd80ad5ae3e5a24da1ac0f791f5308a935a16886f6a WatchSource:0}: Error finding container b1b48bd14c7b60959b504fd80ad5ae3e5a24da1ac0f791f5308a935a16886f6a: Status 404 returned error can't find the container with id b1b48bd14c7b60959b504fd80ad5ae3e5a24da1ac0f791f5308a935a16886f6a
Jul 03 20:37:10.317170 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317148    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7nf6t\""
Jul 03 20:37:10.317253 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317181    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.317253 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317190    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.317253 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317241    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 03 20:37:10.317409 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.317309    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d17de08da950bb8f59f5515256e0343.slice/crio-10fc253b88da8275b35dac23343f55b0d7dc0d17de61d6362bb075bfb0bb24b3 WatchSource:0}: Error finding container 10fc253b88da8275b35dac23343f55b0d7dc0d17de61d6362bb075bfb0bb24b3: Status 404 returned error can't find the container with id 10fc253b88da8275b35dac23343f55b0d7dc0d17de61d6362bb075bfb0bb24b3
Jul 03 20:37:10.317452 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317438    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tlrfb\""
Jul 03 20:37:10.317533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317516    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 03 20:37:10.317592 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.317567    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 03 20:37:10.318341 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.318325    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.318431 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.318409    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:10.318827 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.318804    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-78s8j\""
Jul 03 20:37:10.318827 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.318819    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 03 20:37:10.319000 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.318827    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.319150 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.319134    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.320932 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.320913    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.323717 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.323697    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 03 20:37:10.323782 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.323729    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 03 20:37:10.323979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.323924    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tqqw7\""
Jul 03 20:37:10.323979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.323944    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.324162 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.324144    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 03 20:37:10.326673 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.326656    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:10.326816 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.326715    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:10.326930 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.326912    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdxth\""
Jul 03 20:37:10.326995 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.326963    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 03 20:37:10.327062 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.326992    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 03 20:37:10.329414 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.329385    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:10.329497 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.329443    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:10.372699 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.372681    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 03 20:37:10.394094 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394063    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-kubernetes\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.394229 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394106    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ddbe1-9a74-4630-ba56-313c42396515-multus-daemon-config\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.394229 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394134    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-kubelet\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.394229 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394159    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-os-release\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.394229 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394186    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d98abd91-b06e-4813-a887-a1163be5eefc-host-slash\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.394229 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394215    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsmw\" (UniqueName: \"kubernetes.io/projected/d98abd91-b06e-4813-a887-a1163be5eefc-kube-api-access-wvsmw\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394243    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-cni-bin\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394271    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-registration-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394298    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-dbus\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394332    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-modprobe-d\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394363    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-multus-certs\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394381    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-ovn\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394397    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-lib-modules\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.394453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394433    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d98abd91-b06e-4813-a887-a1163be5eefc-iptables-alerter-script\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394463    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-log-socket\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394491    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovnkube-script-lib\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394508    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e751430-2992-4aeb-b4c3-15660c5f3ffa-host\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394530    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394545    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c68222-8adb-44f7-a47a-029f466be0a3-agent-certs\") pod \"konnectivity-agent-4bgfb\" (UID: \"a3c68222-8adb-44f7-a47a-029f466be0a3\") " pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394567    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-systemd\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394581    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394597    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-etc-kubernetes\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394611    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-slash\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394627    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-system-cni-dir\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394659    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-run\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b0a92a4-5bac-4160-8189-4e689b18c860-tmp\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.394710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394701    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394718    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-cni-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394742    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-conf-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394766    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-systemd-units\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394785    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-node-log\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394799    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysconfig\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394812    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-cnibin\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394836    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cnibin\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394854    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394874    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c68222-8adb-44f7-a47a-029f466be0a3-konnectivity-ca\") pod \"konnectivity-agent-4bgfb\" (UID: \"a3c68222-8adb-44f7-a47a-029f466be0a3\") " pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394889    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-var-lib-kubelet\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394904    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-k8s-cni-cncf-io\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-run-netns\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394964    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovnkube-config\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.394999    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24t6w\" (UniqueName: \"kubernetes.io/projected/f11ddbe1-9a74-4630-ba56-313c42396515-kube-api-access-24t6w\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395033    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395064    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-kube-api-access-vnhlx\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395092    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-sys-fs\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395121    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ddbe1-9a74-4630-ba56-313c42396515-cni-binary-copy\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395145    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-etc-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395164    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-env-overrides\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395186    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-hosts-file\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395227    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwsq\" (UniqueName: \"kubernetes.io/projected/2fa04aaa-635c-4e0b-ba6c-f871684417cb-kube-api-access-6kwsq\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395244    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-cni-multus\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395259    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395292    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstrs\" (UniqueName: \"kubernetes.io/projected/7e751430-2992-4aeb-b4c3-15660c5f3ffa-kube-api-access-hstrs\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395312    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395330    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsnqq\" (UniqueName: \"kubernetes.io/projected/1b0a92a4-5bac-4160-8189-4e689b18c860-kube-api-access-bsnqq\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395350    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-socket-dir-parent\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395367    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-cni-bin\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395415    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovn-node-metrics-cert\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.395861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395430    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8lp\" (UniqueName: \"kubernetes.io/projected/8dd4cc76-ed26-4a30-86db-ba274e4d2925-kube-api-access-fz8lp\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395448    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9kz\" (UniqueName: \"kubernetes.io/projected/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-kube-api-access-tp9kz\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395463    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-hostroot\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-var-lib-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395492    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-cni-netd\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g294z\" (UniqueName: \"kubernetes.io/projected/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-kube-api-access-g294z\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395524    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395553    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-socket-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395572    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-host\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395586    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-systemd\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395606    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395622    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-tuned\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395643    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-system-cni-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395659    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-netns\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395673    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-kubelet\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395688    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-tmp-dir\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.396378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395712    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e751430-2992-4aeb-b4c3-15660c5f3ffa-serviceca\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395744    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-os-release\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395771    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysctl-d\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395794    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysctl-conf\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395810    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-sys\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395825    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-device-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395840    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-etc-selinux\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395878    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-kubelet-config\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.396852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.395903    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.481672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.481596    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 03 20:37:10.496379 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-os-release\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.496438 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496384    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d98abd91-b06e-4813-a887-a1163be5eefc-host-slash\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.496496 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496444    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsmw\" (UniqueName: \"kubernetes.io/projected/d98abd91-b06e-4813-a887-a1163be5eefc-kube-api-access-wvsmw\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.496496 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496471    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-cni-bin\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496483    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d98abd91-b06e-4813-a887-a1163be5eefc-host-slash\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.496602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496516    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-os-release\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.496602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496529    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-cni-bin\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496522    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-registration-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.496602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496571    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-dbus\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.496602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496590    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-modprobe-d\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496609    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-multus-certs\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-ovn\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496644    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-registration-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496657    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-lib-modules\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496680    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d98abd91-b06e-4813-a887-a1163be5eefc-iptables-alerter-script\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496687    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-multus-certs\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496718    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-log-socket\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496728    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-ovn\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496749    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovnkube-script-lib\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496774    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e751430-2992-4aeb-b4c3-15660c5f3ffa-host\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496799    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496811    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-dbus\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496823    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c68222-8adb-44f7-a47a-029f466be0a3-agent-certs\") pod \"konnectivity-agent-4bgfb\" (UID: \"a3c68222-8adb-44f7-a47a-029f466be0a3\") " pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496847    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-lib-modules\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496849    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-systemd\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496891    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-log-socket\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.496902 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496892    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496748    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-modprobe-d\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496894    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-systemd\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496941    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-etc-kubernetes\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496969    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-slash\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.496994    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-system-cni-dir\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497018    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-run\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.497030    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497040    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b0a92a4-5bac-4160-8189-4e689b18c860-tmp\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497068    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.497105    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:10.997083662 +0000 UTC m=+3.077825066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497123    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-cni-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497133    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e751430-2992-4aeb-b4c3-15660c5f3ffa-host\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497154    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-conf-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497168    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-etc-kubernetes\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497208    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-slash\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497216    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-conf-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497257    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-system-cni-dir\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.497807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497301    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-run\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497424    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-cni-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497466    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-systemd-units\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497494    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d98abd91-b06e-4813-a887-a1163be5eefc-iptables-alerter-script\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497536    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-systemd-units\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497494    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovnkube-script-lib\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497551    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-node-log\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497496    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-node-log\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497571    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497591    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysconfig\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-cnibin\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497624    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cnibin\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497640    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c68222-8adb-44f7-a47a-029f466be0a3-konnectivity-ca\") pod \"konnectivity-agent-4bgfb\" (UID: \"a3c68222-8adb-44f7-a47a-029f466be0a3\") " pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497659    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysconfig\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497705    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497720    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cnibin\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497775    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-var-lib-kubelet\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.498724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497806    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-k8s-cni-cncf-io\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497829    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-run-netns\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497872    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovnkube-config\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497910    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-var-lib-kubelet\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497916    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-cnibin\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497925    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24t6w\" (UniqueName: \"kubernetes.io/projected/f11ddbe1-9a74-4630-ba56-313c42396515-kube-api-access-24t6w\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497959    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497968    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-run-netns\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498000    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-kube-api-access-vnhlx\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498014    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498025    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-sys-fs\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498048    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ddbe1-9a74-4630-ba56-313c42396515-cni-binary-copy\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498083    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-etc-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498104    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-env-overrides\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498126    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-hosts-file\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498149    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwsq\" (UniqueName: \"kubernetes.io/projected/2fa04aaa-635c-4e0b-ba6c-f871684417cb-kube-api-access-6kwsq\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498188    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-cni-multus\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.499708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498209    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498232    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstrs\" (UniqueName: \"kubernetes.io/projected/7e751430-2992-4aeb-b4c3-15660c5f3ffa-kube-api-access-hstrs\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498255    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-etc-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498268    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.497957    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-k8s-cni-cncf-io\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsnqq\" (UniqueName: \"kubernetes.io/projected/1b0a92a4-5bac-4160-8189-4e689b18c860-kube-api-access-bsnqq\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498334    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-socket-dir-parent\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498361    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-cni-bin\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498384    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498423    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovn-node-metrics-cert\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498446    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8lp\" (UniqueName: \"kubernetes.io/projected/8dd4cc76-ed26-4a30-86db-ba274e4d2925-kube-api-access-fz8lp\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498467    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9kz\" (UniqueName: \"kubernetes.io/projected/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-kube-api-access-tp9kz\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498473    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovnkube-config\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498478    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498499    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-sys-fs\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-hostroot\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498548    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-cni-multus\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.500645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498555    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-hostroot\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498576    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-var-lib-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498615    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-var-lib-openvswitch\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498625    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-multus-socket-dir-parent\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498645    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-cni-netd\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g294z\" (UniqueName: \"kubernetes.io/projected/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-kube-api-access-g294z\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498704    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498720    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498728    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-socket-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498775    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-host\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498799    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-systemd\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498838    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498867    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-tuned\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498893    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-system-cni-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498943    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-netns\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498969    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-kubelet\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499046    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-env-overrides\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.501571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499070    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-tmp-dir\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499091    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f11ddbe1-9a74-4630-ba56-313c42396515-cni-binary-copy\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499098    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e751430-2992-4aeb-b4c3-15660c5f3ffa-serviceca\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499123    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-hosts-file\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499124    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-os-release\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499187    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysctl-d\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499200    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fa04aaa-635c-4e0b-ba6c-f871684417cb-os-release\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499210    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-socket-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499216    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysctl-conf\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.498334    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3c68222-8adb-44f7-a47a-029f466be0a3-konnectivity-ca\") pod \"konnectivity-agent-4bgfb\" (UID: \"a3c68222-8adb-44f7-a47a-029f466be0a3\") " pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499268    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-host\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499276    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-cni-bin\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499290    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-var-lib-kubelet\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499313    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499324    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499356    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysctl-d\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499371    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499422    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-run-systemd\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.502226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499477    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-system-cni-dir\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499593    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-cni-netd\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499715    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-tmp-dir\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499725    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f11ddbe1-9a74-4630-ba56-313c42396515-host-run-netns\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499805    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-sys\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499834    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-device-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499860    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-etc-selinux\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499911    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e751430-2992-4aeb-b4c3-15660c5f3ffa-serviceca\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499926    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-kubelet-config\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.499953    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500007    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-kubernetes\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500034    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ddbe1-9a74-4630-ba56-313c42396515-multus-daemon-config\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500063    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-kubelet\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500072    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-sys\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500128    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-host-kubelet\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500171    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-device-dir\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500175    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-kubernetes\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500240    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-kubelet-config\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:10.502759 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500248    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8dd4cc76-ed26-4a30-86db-ba274e4d2925-etc-selinux\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.500307    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500336    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-sysctl-conf\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.500365    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:37:11.000348012 +0000 UTC m=+3.081089401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500388    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fa04aaa-635c-4e0b-ba6c-f871684417cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.500969    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f11ddbe1-9a74-4630-ba56-313c42396515-multus-daemon-config\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.501544    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b0a92a4-5bac-4160-8189-4e689b18c860-tmp\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.501664    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3c68222-8adb-44f7-a47a-029f466be0a3-agent-certs\") pod \"konnectivity-agent-4bgfb\" (UID: \"a3c68222-8adb-44f7-a47a-029f466be0a3\") " pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.501987    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b0a92a4-5bac-4160-8189-4e689b18c860-etc-tuned\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.503316 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.502517    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-ovn-node-metrics-cert\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.507180 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.505768    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:10.507180 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.505797    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:10.507180 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.505813    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:10.507180 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:10.505882    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:11.005863917 +0000 UTC m=+3.086605330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:10.507180 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.506389    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsmw\" (UniqueName: \"kubernetes.io/projected/d98abd91-b06e-4813-a887-a1163be5eefc-kube-api-access-wvsmw\") pod \"iptables-alerter-l7rqd\" (UID: \"d98abd91-b06e-4813-a887-a1163be5eefc\") " pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.508029 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.508002    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstrs\" (UniqueName: \"kubernetes.io/projected/7e751430-2992-4aeb-b4c3-15660c5f3ffa-kube-api-access-hstrs\") pod \"node-ca-rcmn4\" (UID: \"7e751430-2992-4aeb-b4c3-15660c5f3ffa\") " pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.509879 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.508616    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwsq\" (UniqueName: \"kubernetes.io/projected/2fa04aaa-635c-4e0b-ba6c-f871684417cb-kube-api-access-6kwsq\") pod \"multus-additional-cni-plugins-jsnzb\" (UID: \"2fa04aaa-635c-4e0b-ba6c-f871684417cb\") " pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.509879 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.508654    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24t6w\" (UniqueName: \"kubernetes.io/projected/f11ddbe1-9a74-4630-ba56-313c42396515-kube-api-access-24t6w\") pod \"multus-9p4vf\" (UID: \"f11ddbe1-9a74-4630-ba56-313c42396515\") " pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.511243 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.511218    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g294z\" (UniqueName: \"kubernetes.io/projected/5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2-kube-api-access-g294z\") pod \"ovnkube-node-l4mn2\" (UID: \"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.511499 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.511420    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8lp\" (UniqueName: \"kubernetes.io/projected/8dd4cc76-ed26-4a30-86db-ba274e4d2925-kube-api-access-fz8lp\") pod \"aws-ebs-csi-driver-node-g77n7\" (UID: \"8dd4cc76-ed26-4a30-86db-ba274e4d2925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.512159 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.512103    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhlx\" (UniqueName: \"kubernetes.io/projected/c84e01a7-e24c-4313-9fb4-636dc71bc1b2-kube-api-access-vnhlx\") pod \"node-resolver-zpjjr\" (UID: \"c84e01a7-e24c-4313-9fb4-636dc71bc1b2\") " pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.512360 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.512337    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsnqq\" (UniqueName: \"kubernetes.io/projected/1b0a92a4-5bac-4160-8189-4e689b18c860-kube-api-access-bsnqq\") pod \"tuned-2tgxt\" (UID: \"1b0a92a4-5bac-4160-8189-4e689b18c860\") " pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.513203 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.513184    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9kz\" (UniqueName: \"kubernetes.io/projected/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-kube-api-access-tp9kz\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:10.626588 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.626552    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9p4vf"
Jul 03 20:37:10.633445 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.633420    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l7rqd"
Jul 03 20:37:10.637374 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.637352    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11ddbe1_9a74_4630_ba56_313c42396515.slice/crio-e72eafd13ecee6546c3de54ea1a2fe1b12073d44f341b5a7617b20082fbb668c WatchSource:0}: Error finding container e72eafd13ecee6546c3de54ea1a2fe1b12073d44f341b5a7617b20082fbb668c: Status 404 returned error can't find the container with id e72eafd13ecee6546c3de54ea1a2fe1b12073d44f341b5a7617b20082fbb668c
Jul 03 20:37:10.640014 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.639986    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:10.644471 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.644453    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpjjr"
Jul 03 20:37:10.644606 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.644580    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98abd91_b06e_4813_a887_a1163be5eefc.slice/crio-3fd717732b443b2fbfe5a138b3924242471ba67542ddd45cdef9c40f12b83df0 WatchSource:0}: Error finding container 3fd717732b443b2fbfe5a138b3924242471ba67542ddd45cdef9c40f12b83df0: Status 404 returned error can't find the container with id 3fd717732b443b2fbfe5a138b3924242471ba67542ddd45cdef9c40f12b83df0
Jul 03 20:37:10.649917 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.649889    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5871fbca_41b9_4cfc_bd4a_4d9d0234b6e2.slice/crio-0e87a1aa078d85b24dbe575fa0be0d5e2a4dbf2bf81456d0a7525d1333b82126 WatchSource:0}: Error finding container 0e87a1aa078d85b24dbe575fa0be0d5e2a4dbf2bf81456d0a7525d1333b82126: Status 404 returned error can't find the container with id 0e87a1aa078d85b24dbe575fa0be0d5e2a4dbf2bf81456d0a7525d1333b82126
Jul 03 20:37:10.650024 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.649914    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rcmn4"
Jul 03 20:37:10.653958 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.653937    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84e01a7_e24c_4313_9fb4_636dc71bc1b2.slice/crio-a406ca2dbf7061b6fd6986c4e436fefdda950ea3bf9a01cd68a5ec62db93d112 WatchSource:0}: Error finding container a406ca2dbf7061b6fd6986c4e436fefdda950ea3bf9a01cd68a5ec62db93d112: Status 404 returned error can't find the container with id a406ca2dbf7061b6fd6986c4e436fefdda950ea3bf9a01cd68a5ec62db93d112
Jul 03 20:37:10.656486 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.656466    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jsnzb"
Jul 03 20:37:10.659677 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.659658    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e751430_2992_4aeb_b4c3_15660c5f3ffa.slice/crio-6ac09e346f89b1e2bc021511968a68d579c6c36cf34f0f56290a3932a7ef3464 WatchSource:0}: Error finding container 6ac09e346f89b1e2bc021511968a68d579c6c36cf34f0f56290a3932a7ef3464: Status 404 returned error can't find the container with id 6ac09e346f89b1e2bc021511968a68d579c6c36cf34f0f56290a3932a7ef3464
Jul 03 20:37:10.661911 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.661892    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7"
Jul 03 20:37:10.665257 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.665226    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa04aaa_635c_4e0b_ba6c_f871684417cb.slice/crio-400808bb07311d6d9898070101902e50efb7e5f40380efefbece5f7b75f01f26 WatchSource:0}: Error finding container 400808bb07311d6d9898070101902e50efb7e5f40380efefbece5f7b75f01f26: Status 404 returned error can't find the container with id 400808bb07311d6d9898070101902e50efb7e5f40380efefbece5f7b75f01f26
Jul 03 20:37:10.669519 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.669471    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:10.671094 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.671070    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd4cc76_ed26_4a30_86db_ba274e4d2925.slice/crio-fbefe59b1e0923b4c33dea0bd130a7895defe960471064ffcc000381d4f21508 WatchSource:0}: Error finding container fbefe59b1e0923b4c33dea0bd130a7895defe960471064ffcc000381d4f21508: Status 404 returned error can't find the container with id fbefe59b1e0923b4c33dea0bd130a7895defe960471064ffcc000381d4f21508
Jul 03 20:37:10.674871 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.674835    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tgxt"
Jul 03 20:37:10.679530 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.679503    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c68222_8adb_44f7_a47a_029f466be0a3.slice/crio-9c8de3974355d9600f5b7ccca11f194e0aaee1a938271685053ada9be535ada5 WatchSource:0}: Error finding container 9c8de3974355d9600f5b7ccca11f194e0aaee1a938271685053ada9be535ada5: Status 404 returned error can't find the container with id 9c8de3974355d9600f5b7ccca11f194e0aaee1a938271685053ada9be535ada5
Jul 03 20:37:10.684455 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:10.684410    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0a92a4_5bac_4160_8189_4e689b18c860.slice/crio-9af183d87bbbc7fe54f83154607e598858e2235a457f3fd780a91450604e8224 WatchSource:0}: Error finding container 9af183d87bbbc7fe54f83154607e598858e2235a457f3fd780a91450604e8224: Status 404 returned error can't find the container with id 9af183d87bbbc7fe54f83154607e598858e2235a457f3fd780a91450604e8224
Jul 03 20:37:10.870739 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.870624    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tgxt" event={"ID":"1b0a92a4-5bac-4160-8189-4e689b18c860","Type":"ContainerStarted","Data":"9af183d87bbbc7fe54f83154607e598858e2235a457f3fd780a91450604e8224"}
Jul 03 20:37:10.872896 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.872819    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4bgfb" event={"ID":"a3c68222-8adb-44f7-a47a-029f466be0a3","Type":"ContainerStarted","Data":"9c8de3974355d9600f5b7ccca11f194e0aaee1a938271685053ada9be535ada5"}
Jul 03 20:37:10.874898 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.874816    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7" event={"ID":"8dd4cc76-ed26-4a30-86db-ba274e4d2925","Type":"ContainerStarted","Data":"fbefe59b1e0923b4c33dea0bd130a7895defe960471064ffcc000381d4f21508"}
Jul 03 20:37:10.877333 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.877236    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerStarted","Data":"400808bb07311d6d9898070101902e50efb7e5f40380efefbece5f7b75f01f26"}
Jul 03 20:37:10.879897 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.879835    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rcmn4" event={"ID":"7e751430-2992-4aeb-b4c3-15660c5f3ffa","Type":"ContainerStarted","Data":"6ac09e346f89b1e2bc021511968a68d579c6c36cf34f0f56290a3932a7ef3464"}
Jul 03 20:37:10.881770 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.881712    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpjjr" event={"ID":"c84e01a7-e24c-4313-9fb4-636dc71bc1b2","Type":"ContainerStarted","Data":"a406ca2dbf7061b6fd6986c4e436fefdda950ea3bf9a01cd68a5ec62db93d112"}
Jul 03 20:37:10.883109 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.883043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal" event={"ID":"fae37a88080d71f4e174849eeb91a89e","Type":"ContainerStarted","Data":"b1b48bd14c7b60959b504fd80ad5ae3e5a24da1ac0f791f5308a935a16886f6a"}
Jul 03 20:37:10.884681 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.884594    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"0e87a1aa078d85b24dbe575fa0be0d5e2a4dbf2bf81456d0a7525d1333b82126"}
Jul 03 20:37:10.886660 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.886577    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l7rqd" event={"ID":"d98abd91-b06e-4813-a887-a1163be5eefc","Type":"ContainerStarted","Data":"3fd717732b443b2fbfe5a138b3924242471ba67542ddd45cdef9c40f12b83df0"}
Jul 03 20:37:10.888338 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.888314    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9p4vf" event={"ID":"f11ddbe1-9a74-4630-ba56-313c42396515","Type":"ContainerStarted","Data":"e72eafd13ecee6546c3de54ea1a2fe1b12073d44f341b5a7617b20082fbb668c"}
Jul 03 20:37:10.889639 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:10.889613    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal" event={"ID":"4d17de08da950bb8f59f5515256e0343","Type":"ContainerStarted","Data":"10fc253b88da8275b35dac23343f55b0d7dc0d17de61d6362bb075bfb0bb24b3"}
Jul 03 20:37:11.003309 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.003272    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:11.003500 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.003337    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:11.003500 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.003460    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:11.003500 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.003473    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:11.003674 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.003542    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:12.003521798 +0000 UTC m=+4.084263187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:11.003674 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.003566    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:37:12.003554753 +0000 UTC m=+4.084296141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:11.104447 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.104405    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:11.104649 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.104602    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:11.104649 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.104627    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:11.104649 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.104641    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:11.104823 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.104699    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:12.104681225 +0000 UTC m=+4.185422616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:11.319647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.319471    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 20:32:10 +0000 UTC" deadline="2028-01-01 01:53:56.633391734 +0000 UTC"
Jul 03 20:37:11.319647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.319534    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13109h16m45.31386183s"
Jul 03 20:37:11.864990 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.864941    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:11.865194 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.865084    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:11.865194 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.865185    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:11.865314 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.865282    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:11.865383 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:11.865342    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:11.865443 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:11.865425    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:12.012188 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:12.012142    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:12.012445 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:12.012222    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:12.012445 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.012375    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:12.012445 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.012445    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:14.012425172 +0000 UTC m=+6.093166563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:12.012886 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.012863    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:12.012997 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.012941    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:37:14.01290777 +0000 UTC m=+6.093649159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:12.112961 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:12.112901    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:12.113183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.113120    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:12.113183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.113156    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:12.113183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.113171    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:12.113350 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:12.113238    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:14.113216787 +0000 UTC m=+6.193958178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:12.320683 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:12.320584    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 20:32:10 +0000 UTC" deadline="2028-04-09 22:39:49.871049862 +0000 UTC"
Jul 03 20:37:12.320683 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:12.320628    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15506h2m37.550426987s"
Jul 03 20:37:13.864269 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:13.864225    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:13.864757 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:13.864380    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:13.864886 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:13.864865    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:13.865022 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:13.865001    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:13.865089 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:13.865076    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:13.865206 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:13.865152    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:14.030727 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:14.030681    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:14.030906 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:14.030754    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:14.030906 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.030893    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:14.031057 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.030978    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:18.030953577 +0000 UTC m=+10.111694970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:14.031057 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.030993    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:14.031189 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.031060    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:37:18.031041471 +0000 UTC m=+10.111782885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:14.131826 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:14.131741    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:14.132012 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.131915    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:14.132012 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.131950    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:14.132012 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.131964    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:14.132207 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:14.132023    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:18.132003299 +0000 UTC m=+10.212744708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:15.865306 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:15.864874    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:15.865306 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:15.865026    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:15.865306 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:15.865031    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:15.865306 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:15.865119    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:15.865306 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:15.865165    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:15.865306 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:15.865231    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:17.864355 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:17.864316    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:17.864843 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:17.864469    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:17.864843 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:17.864469    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:17.864843 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:17.864599    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:17.864843 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:17.864634    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:17.864843 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:17.864756    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:18.066469 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:18.066394    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:18.066630 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:18.066517    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:18.066703 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.066636    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:18.066749 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.066703    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:37:26.066682345 +0000 UTC m=+18.147423734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:18.067004 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.066852    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:18.067004 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.066975    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:26.066955954 +0000 UTC m=+18.147697340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:18.167439 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:18.167208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:18.167589 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.167441    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:18.167589 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.167460    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:18.167589 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.167475    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:18.167589 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:18.167535    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:26.167517932 +0000 UTC m=+18.248259321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:19.864564 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:19.864522    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:19.864564 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:19.864558    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:19.865077 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:19.864661    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:19.865077 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:19.864719    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:19.865077 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:19.864731    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:19.865077 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:19.864832    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:21.864463 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:21.864429    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:21.864913 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:21.864472    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:21.864913 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:21.864533    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:21.864913 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:21.864551    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:21.864913 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:21.864635    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:21.864913 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:21.864711    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:23.864696 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:23.864654    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:23.864696 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:23.864654    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:23.864696 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:23.864680    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:23.865268 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:23.864791    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:23.865268 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:23.864834    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:23.865268 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:23.864976    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:25.864437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:25.864337    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:25.864437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:25.864382    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:25.864437 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:25.864382    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:25.865020 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:25.864464    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:25.865020 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:25.864574    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:25.865020 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:25.864665    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:26.129444 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:26.129361    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:26.129444 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:26.129405    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:26.129662 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.129502    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:26.129662 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.129515    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:26.129662 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.129555    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:42.129537298 +0000 UTC m=+34.210278687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:26.129662 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.129570    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:37:42.129562958 +0000 UTC m=+34.210304343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:26.230059 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:26.230016    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:26.230243 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.230188    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:26.230243 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.230211    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:26.230243 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.230223    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:26.230444 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:26.230285    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:37:42.230265244 +0000 UTC m=+34.311006630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:27.864256 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:27.864221    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:27.864256 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:27.864253    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:27.864789 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:27.864347    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:27.864789 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:27.864406    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:27.864789 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:27.864512    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:27.864789 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:27.864606    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:29.864146 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:29.864105    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:29.864146 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:29.864146    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:29.864663 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:29.864159    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:29.864663 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:29.864249    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:29.864663 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:29.864395    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:29.864663 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:29.864511    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:31.864709 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.864523    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:31.865181 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.864561    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:31.865181 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:31.864793    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:31.865181 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.864570    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:31.865181 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:31.864856    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:31.865181 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:31.864963    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:31.945342 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.945310    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal" event={"ID":"4d17de08da950bb8f59f5515256e0343","Type":"ContainerStarted","Data":"81dcdc72f810554646e58c126976e5748e19af405a90f28a30d4fd90b9a3327a"}
Jul 03 20:37:31.949477 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.949454    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tgxt" event={"ID":"1b0a92a4-5bac-4160-8189-4e689b18c860","Type":"ContainerStarted","Data":"ab4ed6aa42398f782d62b2bdb8451616c87160fb4465bdfcbddcb5ca28460473"}
Jul 03 20:37:31.952517 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.952491    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"67d91a983a2b9bafe1aaf05be0204a08362b73864124b902fbdd09fe9389506d"}
Jul 03 20:37:31.952611 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.952553    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"d5431cac34468d4d0599d4196511a0740ac7c7fb223341b30a751bc4d1b935e5"}
Jul 03 20:37:31.952611 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.952584    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"cbe4a7a7050515d004c8e40cc1908c5e2aeadf628525f30e147269cf111aa37c"}
Jul 03 20:37:31.952720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.952614    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"df5799efcbc2a93b5847d198ab3e4463e5deb56f52702867ed5666a6de5c5545"}
Jul 03 20:37:31.952720 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.952647    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"39bf09493282bc00ce1655b37cd9af449f54d58b16c1bc2ac6295cc3857ca29e"}
Jul 03 20:37:31.953834 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.953813    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9p4vf" event={"ID":"f11ddbe1-9a74-4630-ba56-313c42396515","Type":"ContainerStarted","Data":"c01e4f34b2d9ec2c903e16629d9539e2dbc7c4ab556ccfeafedd53275d71fbae"}
Jul 03 20:37:31.962773 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.962732    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-236.ec2.internal" podStartSLOduration=21.96272134 podStartE2EDuration="21.96272134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:31.96269689 +0000 UTC m=+24.043438297" watchObservedRunningTime="2026-07-03 20:37:31.96272134 +0000 UTC m=+24.043462747"
Jul 03 20:37:31.980039 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.979990    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tgxt" podStartSLOduration=2.558185494 podStartE2EDuration="22.979972645s" totalImagesPullingTime="20.421787151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:31.97941901 +0000 UTC m=+24.060160419" watchObservedRunningTime="2026-07-03 20:37:31.979972645 +0000 UTC m=+24.060714054"
Jul 03 20:37:31.997419 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:31.997367    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9p4vf" podStartSLOduration=2.499021861 podStartE2EDuration="22.997349026s" totalImagesPullingTime="20.498327165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:31.996752973 +0000 UTC m=+24.077494380" watchObservedRunningTime="2026-07-03 20:37:31.997349026 +0000 UTC m=+24.078090435"
Jul 03 20:37:32.958019 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.957749    2553 generic.go:356] "Generic (PLEG): container finished" podID="2fa04aaa-635c-4e0b-ba6c-f871684417cb" containerID="cf340695bc0f94fa754cb5fcd90c3ec2638601fe98638226f94b086b53ed6fb5" exitCode=0
Jul 03 20:37:32.958732 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.957837    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerDied","Data":"cf340695bc0f94fa754cb5fcd90c3ec2638601fe98638226f94b086b53ed6fb5"}
Jul 03 20:37:32.959782 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.959757    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rcmn4" event={"ID":"7e751430-2992-4aeb-b4c3-15660c5f3ffa","Type":"ContainerStarted","Data":"3b21ac48f4d5fc1bcfc4a9b77ce899c4a709c11a26b0272276dad0a09ece0781"}
Jul 03 20:37:32.961752 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.961727    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpjjr" event={"ID":"c84e01a7-e24c-4313-9fb4-636dc71bc1b2","Type":"ContainerStarted","Data":"a2fb574dd8a7762efcb2e99153695d022524610f0896a73f44d313e610c19a6d"}
Jul 03 20:37:32.963288 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.963266    2553 generic.go:356] "Generic (PLEG): container finished" podID="fae37a88080d71f4e174849eeb91a89e" containerID="aa81a711244377a54b276473c707a92963b1bdf7cf98495e05d0eb014845dab6" exitCode=0
Jul 03 20:37:32.963384 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.963343    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal" event={"ID":"fae37a88080d71f4e174849eeb91a89e","Type":"ContainerDied","Data":"aa81a711244377a54b276473c707a92963b1bdf7cf98495e05d0eb014845dab6"}
Jul 03 20:37:32.966863 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.966842    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"a7cb740ecdda1cdf682aed154f9f584790de770e2ea0e8b3d3a34162d36347f8"}
Jul 03 20:37:32.968450 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.968427    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l7rqd" event={"ID":"d98abd91-b06e-4813-a887-a1163be5eefc","Type":"ContainerStarted","Data":"487dfc3720d71efeaf19276a9386ca5993d7924ba50e11b7e846e493d2d1f866"}
Jul 03 20:37:32.970036 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.970001    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4bgfb" event={"ID":"a3c68222-8adb-44f7-a47a-029f466be0a3","Type":"ContainerStarted","Data":"33772d5b04cfb5e1df69d3f96afd07b9e6e8573df0603d0f8028d3fdd397636e"}
Jul 03 20:37:32.971673 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:32.971648    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7" event={"ID":"8dd4cc76-ed26-4a30-86db-ba274e4d2925","Type":"ContainerStarted","Data":"a4ee8cccc37ad532a6f155befd2efcbfacfd420e3dedb33dfb11ff2d0d649b85"}
Jul 03 20:37:33.008990 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.008944    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l7rqd" podStartSLOduration=3.549966579 podStartE2EDuration="24.008926447s" totalImagesPullingTime="20.458959868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:32.995675839 +0000 UTC m=+25.076417247" watchObservedRunningTime="2026-07-03 20:37:33.008926447 +0000 UTC m=+25.089667857"
Jul 03 20:37:33.009353 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.009319    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rcmn4" podStartSLOduration=3.566379214 podStartE2EDuration="24.009309334s" totalImagesPullingTime="20.44293012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:33.008927602 +0000 UTC m=+25.089669009" watchObservedRunningTime="2026-07-03 20:37:33.009309334 +0000 UTC m=+25.090050800"
Jul 03 20:37:33.025909 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.025855    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4bgfb" podStartSLOduration=3.630354314 podStartE2EDuration="24.025838129s" totalImagesPullingTime="20.395483815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:33.025507709 +0000 UTC m=+25.106249118" watchObservedRunningTime="2026-07-03 20:37:33.025838129 +0000 UTC m=+25.106579538"
Jul 03 20:37:33.055611 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.055566    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zpjjr" podStartSLOduration=3.607174399 podStartE2EDuration="24.055551624s" totalImagesPullingTime="20.448377225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:33.05505711 +0000 UTC m=+25.135798532" watchObservedRunningTime="2026-07-03 20:37:33.055551624 +0000 UTC m=+25.136293032"
Jul 03 20:37:33.324884 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.324850    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 03 20:37:33.720377 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.720261    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-03T20:37:33.32487954Z","UUID":"af7edcef-6696-4552-8417-36191a5e73ce","Handler":null,"Name":"","Endpoint":""}
Jul 03 20:37:33.721887 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.721864    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 03 20:37:33.722046 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.721913    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 03 20:37:33.864303 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.864276    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:33.864303 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.864293    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:33.864530 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.864281    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:33.864530 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:33.864421    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:33.864530 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:33.864485    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:33.864679 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:33.864565    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:33.976343 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.976260    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal" event={"ID":"fae37a88080d71f4e174849eeb91a89e","Type":"ContainerStarted","Data":"64f855256c9c4300bc2df866e5d4b883f612b9533011eb09f8df04e7ee59dfd2"}
Jul 03 20:37:33.978998 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.978758    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7" event={"ID":"8dd4cc76-ed26-4a30-86db-ba274e4d2925","Type":"ContainerStarted","Data":"23cb2dd38122f57b9b372bd4c212c1eac7c37ca7830e6af97b2fb823857fca4a"}
Jul 03 20:37:33.992919 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:33.992860    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-236.ec2.internal" podStartSLOduration=23.992846714 podStartE2EDuration="23.992846714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:33.992596018 +0000 UTC m=+26.073337427" watchObservedRunningTime="2026-07-03 20:37:33.992846714 +0000 UTC m=+26.073588121"
Jul 03 20:37:34.982713 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:34.982626    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7" event={"ID":"8dd4cc76-ed26-4a30-86db-ba274e4d2925","Type":"ContainerStarted","Data":"8a1b1dd0834deff8378a342b4dae4b1f9673a51011a39177535aed9e2feb69d3"}
Jul 03 20:37:34.985970 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:34.985940    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"9ece863ca0642ede1dbb0e8bd09522607e549192e031a19699f77535ceb2143a"}
Jul 03 20:37:35.002686 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:35.002642    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g77n7" podStartSLOduration=3.413027599 podStartE2EDuration="26.002622598s" totalImagesPullingTime="22.589594999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:35.002039965 +0000 UTC m=+27.082781374" watchObservedRunningTime="2026-07-03 20:37:35.002622598 +0000 UTC m=+27.083364007"
Jul 03 20:37:35.864991 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:35.864945    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:35.865148 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:35.864963    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:35.865148 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:35.865090    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:35.865148 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:35.864967    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:35.865148 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:35.865125    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:35.865336 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:35.865237    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:36.099702 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:36.099671    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:36.100328 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:36.100304    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:36.991501 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:36.991459    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:36.992058 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:36.992040    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4bgfb"
Jul 03 20:37:37.864327 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.864170    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:37.864937 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.864170    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:37.864937 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:37.864412    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:37.864937 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.864189    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:37.864937 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:37.864465    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:37.864937 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:37.864583    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:37.995028 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.994998    2553 generic.go:356] "Generic (PLEG): container finished" podID="2fa04aaa-635c-4e0b-ba6c-f871684417cb" containerID="0bbf5c52a9be3017005ff6528bcc33e811fabea5db929f0796a566506051afee" exitCode=0
Jul 03 20:37:37.995131 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.995076    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerDied","Data":"0bbf5c52a9be3017005ff6528bcc33e811fabea5db929f0796a566506051afee"}
Jul 03 20:37:37.998831 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.998810    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" event={"ID":"5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2","Type":"ContainerStarted","Data":"07079c833b52bfbb4935cfe728eb769d07b57d7905630752033dc90f2bbcc738"}
Jul 03 20:37:37.999106 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.999087    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:37.999207 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:37.999163    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:38.013741 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:38.013693    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:38.047614 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:38.047570    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2" podStartSLOduration=8.322329508 podStartE2EDuration="29.047556476s" totalImagesPullingTime="20.725226968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:38.047110267 +0000 UTC m=+30.127851676" watchObservedRunningTime="2026-07-03 20:37:38.047556476 +0000 UTC m=+30.128297883"
Jul 03 20:37:39.001546 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.001519    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:39.015458 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.015431    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:37:39.514609 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.514325    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xwkdt"]
Jul 03 20:37:39.514780 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.514719    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:39.514850 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:39.514832    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:39.514933 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.514911    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qf5hm"]
Jul 03 20:37:39.515015 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.515004    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:39.515109 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:39.515092    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:39.516793 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.516767    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ts9j8"]
Jul 03 20:37:39.516921 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:39.516852    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:39.517018 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:39.516952    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:40.005548 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:40.005512    2553 generic.go:356] "Generic (PLEG): container finished" podID="2fa04aaa-635c-4e0b-ba6c-f871684417cb" containerID="a04114cb45ff5f89c67aa98c7e583e0e81dc701f51005faf1f8f76d73275b4c9" exitCode=0
Jul 03 20:37:40.005997 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:40.005591    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerDied","Data":"a04114cb45ff5f89c67aa98c7e583e0e81dc701f51005faf1f8f76d73275b4c9"}
Jul 03 20:37:40.864854 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:40.864771    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:40.865021 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:40.864922    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:40.865021 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:40.864924    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:40.865021 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:40.864969    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:40.865176 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:40.865039    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:40.865176 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:40.865118    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:42.012282 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.012200    2553 generic.go:356] "Generic (PLEG): container finished" podID="2fa04aaa-635c-4e0b-ba6c-f871684417cb" containerID="f2c8b57c1d071b8a71138d6b75956a791060cd581bfb12410d5ec7621fcac499" exitCode=0
Jul 03 20:37:42.012282 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.012252    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerDied","Data":"f2c8b57c1d071b8a71138d6b75956a791060cd581bfb12410d5ec7621fcac499"}
Jul 03 20:37:42.135047 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.135012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:42.135226 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.135075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:42.135226 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.135159    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:42.135226 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.135164    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:42.135381 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.135246    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret podName:55d54b63-00e7-4ef2-b36b-0cc61d1a434c nodeName:}" failed. No retries permitted until 2026-07-03 20:38:14.135224702 +0000 UTC m=+66.215966104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret") pod "global-pull-secret-syncer-xwkdt" (UID: "55d54b63-00e7-4ef2-b36b-0cc61d1a434c") : object "kube-system"/"original-pull-secret" not registered
Jul 03 20:37:42.135381 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.135270    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:38:14.135257635 +0000 UTC m=+66.215999024 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 03 20:37:42.235634 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.235602    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:42.235753 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.235729    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 03 20:37:42.235753 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.235743    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 03 20:37:42.235753 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.235753    2553 projected.go:196] Error preparing data for projected volume kube-api-access-tlsxb for pod openshift-network-diagnostics/network-check-target-qf5hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:42.235857 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.235804    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb podName:3550f492-132d-4c82-8aa6-677314d78fa7 nodeName:}" failed. No retries permitted until 2026-07-03 20:38:14.235791519 +0000 UTC m=+66.316532905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tlsxb" (UniqueName: "kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb") pod "network-check-target-qf5hm" (UID: "3550f492-132d-4c82-8aa6-677314d78fa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 03 20:37:42.864281 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.864199    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:42.864281 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.864219    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:42.864495 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:42.864209    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:42.864495 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.864332    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xwkdt" podUID="55d54b63-00e7-4ef2-b36b-0cc61d1a434c"
Jul 03 20:37:42.864495 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.864466    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:37:42.864667 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:42.864582    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qf5hm" podUID="3550f492-132d-4c82-8aa6-677314d78fa7"
Jul 03 20:37:44.229495 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.229464    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-236.ec2.internal" event="NodeReady"
Jul 03 20:37:44.229918 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.229600    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 03 20:37:44.298658 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.298625    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pz4zv"]
Jul 03 20:37:44.302612 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.302418    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.302612 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.302608    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6nz9p"]
Jul 03 20:37:44.306227 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.306207    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.308044 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.308026    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v2wzp\""
Jul 03 20:37:44.308138 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.308065    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 03 20:37:44.308138 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.308076    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 03 20:37:44.310411 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.310384    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 03 20:37:44.310712 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.310693    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 03 20:37:44.310835 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.310819    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 03 20:37:44.310961 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.310935    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lzmlt\""
Jul 03 20:37:44.320805 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.320762    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nz9p"]
Jul 03 20:37:44.322456 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.322434    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pz4zv"]
Jul 03 20:37:44.450514 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.450481    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69053652-4d62-46a7-b20e-7a963f70b45d-config-volume\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.450674 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.450531    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.450674 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.450555    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becd33cb-180b-42fe-bccb-161a55b71620-cert\") pod \"ingress-canary-6nz9p\" (UID: \"becd33cb-180b-42fe-bccb-161a55b71620\") " pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.450674 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.450661    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdfr\" (UniqueName: \"kubernetes.io/projected/becd33cb-180b-42fe-bccb-161a55b71620-kube-api-access-2zdfr\") pod \"ingress-canary-6nz9p\" (UID: \"becd33cb-180b-42fe-bccb-161a55b71620\") " pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.450847 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.450697    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpnf\" (UniqueName: \"kubernetes.io/projected/69053652-4d62-46a7-b20e-7a963f70b45d-kube-api-access-4vpnf\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.450847 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.450761    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69053652-4d62-46a7-b20e-7a963f70b45d-tmp-dir\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.551791 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.551760    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.551979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.551799    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becd33cb-180b-42fe-bccb-161a55b71620-cert\") pod \"ingress-canary-6nz9p\" (UID: \"becd33cb-180b-42fe-bccb-161a55b71620\") " pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.551979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.551829    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdfr\" (UniqueName: \"kubernetes.io/projected/becd33cb-180b-42fe-bccb-161a55b71620-kube-api-access-2zdfr\") pod \"ingress-canary-6nz9p\" (UID: \"becd33cb-180b-42fe-bccb-161a55b71620\") " pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.551979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.551852    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpnf\" (UniqueName: \"kubernetes.io/projected/69053652-4d62-46a7-b20e-7a963f70b45d-kube-api-access-4vpnf\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.551979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.551909    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69053652-4d62-46a7-b20e-7a963f70b45d-tmp-dir\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.551979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.551950    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69053652-4d62-46a7-b20e-7a963f70b45d-config-volume\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.551979 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:44.551952    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:37:44.552259 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:44.552029    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:37:45.052007267 +0000 UTC m=+37.132748659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:37:44.552621 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.552585    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69053652-4d62-46a7-b20e-7a963f70b45d-tmp-dir\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.552860 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.552832    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69053652-4d62-46a7-b20e-7a963f70b45d-config-volume\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.556819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.556791    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/becd33cb-180b-42fe-bccb-161a55b71620-cert\") pod \"ingress-canary-6nz9p\" (UID: \"becd33cb-180b-42fe-bccb-161a55b71620\") " pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.562581 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.562562    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpnf\" (UniqueName: \"kubernetes.io/projected/69053652-4d62-46a7-b20e-7a963f70b45d-kube-api-access-4vpnf\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:44.563134 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.563115    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdfr\" (UniqueName: \"kubernetes.io/projected/becd33cb-180b-42fe-bccb-161a55b71620-kube-api-access-2zdfr\") pod \"ingress-canary-6nz9p\" (UID: \"becd33cb-180b-42fe-bccb-161a55b71620\") " pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.622188 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.622163    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nz9p"
Jul 03 20:37:44.701581 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.701529    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:44.777357 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.777305    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nz9p"]
Jul 03 20:37:44.782012 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:44.781982    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecd33cb_180b_42fe_bccb_161a55b71620.slice/crio-718e20c59a8a25204e10faed5cde81b2dc7251b43dfb3ad0e431b2db3d948295 WatchSource:0}: Error finding container 718e20c59a8a25204e10faed5cde81b2dc7251b43dfb3ad0e431b2db3d948295: Status 404 returned error can't find the container with id 718e20c59a8a25204e10faed5cde81b2dc7251b43dfb3ad0e431b2db3d948295
Jul 03 20:37:44.803820 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.803756    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:44.861178 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.861128    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:44.864456 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.864429    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:37:44.864591 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.864440    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:37:44.864591 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.864566    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:37:44.867448 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.867427    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 03 20:37:44.867448 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.867440    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 03 20:37:44.868608 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.868588    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ml64n\""
Jul 03 20:37:44.868608 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.868599    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4xwrm\""
Jul 03 20:37:44.868767 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.868639    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 03 20:37:44.868767 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:44.868761    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 03 20:37:45.021306 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:45.021272    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nz9p" event={"ID":"becd33cb-180b-42fe-bccb-161a55b71620","Type":"ContainerStarted","Data":"718e20c59a8a25204e10faed5cde81b2dc7251b43dfb3ad0e431b2db3d948295"}
Jul 03 20:37:45.057362 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:45.057278    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:45.057503 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:45.057407    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:37:45.057503 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:45.057462    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:37:46.057443853 +0000 UTC m=+38.138185256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:37:45.884290 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:45.884250    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:46.065632 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:46.065548    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:46.065794 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:46.065672    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:37:46.065794 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:46.065739    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:37:48.065724214 +0000 UTC m=+40.146465601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:37:47.030169 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:47.029851    2553 generic.go:356] "Generic (PLEG): container finished" podID="2fa04aaa-635c-4e0b-ba6c-f871684417cb" containerID="efe2319aa7ce6aee9162fa3bfdffc0eee0e80ec85d66c637eb9e6a66e3aa4498" exitCode=0
Jul 03 20:37:47.030169 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:47.029952    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerDied","Data":"efe2319aa7ce6aee9162fa3bfdffc0eee0e80ec85d66c637eb9e6a66e3aa4498"}
Jul 03 20:37:48.036828 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:48.036796    2553 generic.go:356] "Generic (PLEG): container finished" podID="2fa04aaa-635c-4e0b-ba6c-f871684417cb" containerID="bad250793f5950b8b1db38f2adab6bf76621562e1010e7a525ffdd943dac9feb" exitCode=0
Jul 03 20:37:48.037257 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:48.036837    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerDied","Data":"bad250793f5950b8b1db38f2adab6bf76621562e1010e7a525ffdd943dac9feb"}
Jul 03 20:37:48.081776 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:48.081745    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:48.081948 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:48.081888    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:37:48.082012 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:48.081979    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:37:52.081955259 +0000 UTC m=+44.162696657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:37:48.561560 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:48.561527    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:48.662216 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:48.662137    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:49.040504 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.040473    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nz9p" event={"ID":"becd33cb-180b-42fe-bccb-161a55b71620","Type":"ContainerStarted","Data":"412eeaa32ec3f3c7490fbe18978dc5903d8537f27418ef3aad209b943b0b425e"}
Jul 03 20:37:49.043840 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.043815    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" event={"ID":"2fa04aaa-635c-4e0b-ba6c-f871684417cb","Type":"ContainerStarted","Data":"67754c19c6b15fb6dcc67ca6df726d3ae702964f98466e81ed9fe18a3b254eda"}
Jul 03 20:37:49.056950 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.056887    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6nz9p" podStartSLOduration=1.164798784 podStartE2EDuration="5.056873635s" totalImagesPullingTime="3.892074851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:49.056364925 +0000 UTC m=+41.137106333" watchObservedRunningTime="2026-07-03 20:37:49.056873635 +0000 UTC m=+41.137615043"
Jul 03 20:37:49.081083 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.081038    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jsnzb" podStartSLOduration=7.862384627 podStartE2EDuration="40.081022947s" totalImagesPullingTime="32.21863832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:37:49.079246713 +0000 UTC m=+41.159988121" watchObservedRunningTime="2026-07-03 20:37:49.081022947 +0000 UTC m=+41.161764355"
Jul 03 20:37:49.200753 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.200718    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:49.300567 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.300492    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:49.388472 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:49.388441    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:50.513130 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:50.513099    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:50.568107 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:50.568073    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:52.109414 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:52.109381    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:37:52.109782 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:52.109523    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:37:52.109782 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:37:52.109579    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:38:00.109562075 +0000 UTC m=+52.190303479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:37:53.762325 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:53.762293    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:53.814423 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:53.814395    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:54.340559 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.340530    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:54.396825 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.396798    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:54.686980 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.686884    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"]
Jul 03 20:37:54.709680 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.709655    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"]
Jul 03 20:37:54.709810 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.709777    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.712625 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.712600    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 03 20:37:54.712760 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.712600    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 03 20:37:54.712760 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.712608    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 03 20:37:54.712760 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.712655    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 03 20:37:54.721468 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.721447    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"]
Jul 03 20:37:54.721563 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.721482    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"]
Jul 03 20:37:54.721597 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.721559    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.724246 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.724223    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 03 20:37:54.724421 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.724405    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 03 20:37:54.724991 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.724973    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 03 20:37:54.725044 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.725004    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 03 20:37:54.827699 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827665    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-klusterlet-config\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.827699 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827697    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-hub\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827751    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827777    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-ca\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827803    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hz27\" (UniqueName: \"kubernetes.io/projected/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-kube-api-access-7hz27\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827857    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827888    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgtv\" (UniqueName: \"kubernetes.io/projected/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-kube-api-access-prgtv\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827932    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-tmp\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.828127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.827952    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.928957 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.928924    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.929105 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.928972    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgtv\" (UniqueName: \"kubernetes.io/projected/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-kube-api-access-prgtv\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.929105 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.928992    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-tmp\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.929105 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.929105 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929051    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-klusterlet-config\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.929105 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929077    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-hub\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.929366 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929131    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.929366 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929171    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-ca\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.929366 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929217    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hz27\" (UniqueName: \"kubernetes.io/projected/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-kube-api-access-7hz27\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.929507 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929424    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-tmp\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.929795 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.929765    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.933192 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.933163    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-hub\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.933192 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.933181    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-ca\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.933372 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.933262    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.933372 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.933319    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-klusterlet-config\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.933372 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.933327    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:54.936667 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.936643    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgtv\" (UniqueName: \"kubernetes.io/projected/d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3-kube-api-access-prgtv\") pod \"klusterlet-addon-workmgr-585dc499df-jh4dq\" (UID: \"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:54.936751 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:54.936735    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hz27\" (UniqueName: \"kubernetes.io/projected/407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5-kube-api-access-7hz27\") pod \"cluster-proxy-proxy-agent-548dfc4744-4w9fl\" (UID: \"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:55.019333 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:55.019256    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:37:55.040568 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:55.040536    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:37:55.193083 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:55.193043    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"]
Jul 03 20:37:55.197712 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:55.197688    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2736f21_ac43_42e5_b3c1_2aeb54f5c3d3.slice/crio-6b09a56d3b8f64c18405cdce3d0f01a75223c136ad91c81f6b7cff8a08d30eba WatchSource:0}: Error finding container 6b09a56d3b8f64c18405cdce3d0f01a75223c136ad91c81f6b7cff8a08d30eba: Status 404 returned error can't find the container with id 6b09a56d3b8f64c18405cdce3d0f01a75223c136ad91c81f6b7cff8a08d30eba
Jul 03 20:37:55.211253 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:55.211212    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"]
Jul 03 20:37:55.214227 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:37:55.214198    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod407f9e13_1ebd_4f81_a4b7_0e3b0e73b0f5.slice/crio-9c307ef3ea555bbbcd4e2255721d270a4058dc00ecd48f65d838abd1904b9f0d WatchSource:0}: Error finding container 9c307ef3ea555bbbcd4e2255721d270a4058dc00ecd48f65d838abd1904b9f0d: Status 404 returned error can't find the container with id 9c307ef3ea555bbbcd4e2255721d270a4058dc00ecd48f65d838abd1904b9f0d
Jul 03 20:37:56.061801 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:56.061766    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" event={"ID":"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5","Type":"ContainerStarted","Data":"9c307ef3ea555bbbcd4e2255721d270a4058dc00ecd48f65d838abd1904b9f0d"}
Jul 03 20:37:56.063371 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:56.063335    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" event={"ID":"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3","Type":"ContainerStarted","Data":"6b09a56d3b8f64c18405cdce3d0f01a75223c136ad91c81f6b7cff8a08d30eba"}
Jul 03 20:37:59.687297 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:59.687262    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:37:59.746491 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:37:59.746456    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:00.172647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:00.172611    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:38:00.172818 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:00.172788    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:38:00.172884 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:00.172872    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:38:16.172847481 +0000 UTC m=+68.253588885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:38:02.080827 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.080787    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" event={"ID":"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3","Type":"ContainerStarted","Data":"bc7a0eac0afc42c843d40ce0f221c048fc70437065a1bfa5ef1a711039f8e99d"}
Jul 03 20:38:02.081342 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.080930    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:38:02.082870 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.082850    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:38:02.083032 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.083016    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" event={"ID":"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5","Type":"ContainerStarted","Data":"cd00c218ab8ecf7452e2db3e903a0f12d35f25796126eb8e3463497019576bfd"}
Jul 03 20:38:02.083085 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.083041    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" event={"ID":"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5","Type":"ContainerStarted","Data":"40ae03bf87b1053475a767db264f07cd3ceea1188d429dccd600043b7dcffd4a"}
Jul 03 20:38:02.083085 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.083055    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" event={"ID":"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5","Type":"ContainerStarted","Data":"e1e1ad2ccc73e20c82407ee926dbf02b4b92c75c9049b50f4022a4e4f92f56ab"}
Jul 03 20:38:02.097042 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.096997    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" podStartSLOduration=2.155536645 podStartE2EDuration="8.096983632s" totalImagesPullingTime="5.941446987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:38:02.096539519 +0000 UTC m=+54.177280928" watchObservedRunningTime="2026-07-03 20:38:02.096983632 +0000 UTC m=+54.177725039"
Jul 03 20:38:02.114459 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:02.114405    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" podStartSLOduration=2.205056463 podStartE2EDuration="8.114389453s" totalImagesPullingTime="5.90933299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:38:02.114168598 +0000 UTC m=+54.194910013" watchObservedRunningTime="2026-07-03 20:38:02.114389453 +0000 UTC m=+54.195130862"
Jul 03 20:38:11.018015 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:11.017982    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l4mn2"
Jul 03 20:38:11.172844 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:11.172811    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:11.239484 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:11.239452    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:11.302325 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:11.302292    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:11.495086 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:11.495050    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:11.604672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:11.604583    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:14.169159 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.169113    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:38:14.169159 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.169163    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:38:14.172106 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.172088    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 03 20:38:14.172174 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.172092    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 03 20:38:14.179619 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:14.179595    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 03 20:38:14.179724 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:14.179670    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:39:18.179646825 +0000 UTC m=+130.260388213 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : secret "metrics-daemon-secret" not found
Jul 03 20:38:14.181684 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.181666    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/55d54b63-00e7-4ef2-b36b-0cc61d1a434c-original-pull-secret\") pod \"global-pull-secret-syncer-xwkdt\" (UID: \"55d54b63-00e7-4ef2-b36b-0cc61d1a434c\") " pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:38:14.270307 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.270271    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:38:14.273120 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.273098    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 03 20:38:14.283682 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.283660    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 03 20:38:14.290031 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.290003    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xwkdt"
Jul 03 20:38:14.293712 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.293690    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlsxb\" (UniqueName: \"kubernetes.io/projected/3550f492-132d-4c82-8aa6-677314d78fa7-kube-api-access-tlsxb\") pod \"network-check-target-qf5hm\" (UID: \"3550f492-132d-4c82-8aa6-677314d78fa7\") " pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:38:14.405326 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.405296    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xwkdt"]
Jul 03 20:38:14.408322 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:38:14.408291    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d54b63_00e7_4ef2_b36b_0cc61d1a434c.slice/crio-d417ab260064561117f80ac959f294a7eb0b0cee16fe5b0a735da366ecb9436e WatchSource:0}: Error finding container d417ab260064561117f80ac959f294a7eb0b0cee16fe5b0a735da366ecb9436e: Status 404 returned error can't find the container with id d417ab260064561117f80ac959f294a7eb0b0cee16fe5b0a735da366ecb9436e
Jul 03 20:38:14.586872 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.586844    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ml64n\""
Jul 03 20:38:14.594333 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.594315    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:38:14.714001 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:14.713972    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qf5hm"]
Jul 03 20:38:14.716595 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:38:14.716561    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3550f492_132d_4c82_8aa6_677314d78fa7.slice/crio-74abe35b04075e684b9445ff921e6044e610615a7e669df9810319d1c1c2c950 WatchSource:0}: Error finding container 74abe35b04075e684b9445ff921e6044e610615a7e669df9810319d1c1c2c950: Status 404 returned error can't find the container with id 74abe35b04075e684b9445ff921e6044e610615a7e669df9810319d1c1c2c950
Jul 03 20:38:15.118086 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:15.118043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qf5hm" event={"ID":"3550f492-132d-4c82-8aa6-677314d78fa7","Type":"ContainerStarted","Data":"74abe35b04075e684b9445ff921e6044e610615a7e669df9810319d1c1c2c950"}
Jul 03 20:38:15.119553 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:15.119521    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xwkdt" event={"ID":"55d54b63-00e7-4ef2-b36b-0cc61d1a434c","Type":"ContainerStarted","Data":"d417ab260064561117f80ac959f294a7eb0b0cee16fe5b0a735da366ecb9436e"}
Jul 03 20:38:16.184749 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:16.184651    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:38:16.185220 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:16.184818    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:38:16.185220 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:16.184905    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:38:48.184869049 +0000 UTC m=+100.265610440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:38:16.706963 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:16.706916    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:16.770651 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:16.770618    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:20.134769 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:20.134728    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qf5hm" event={"ID":"3550f492-132d-4c82-8aa6-677314d78fa7","Type":"ContainerStarted","Data":"4fcbf9946fe585a858d7334805559516df3f36b86e3e4d16c7c07f66d5a19f53"}
Jul 03 20:38:20.135224 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:20.134846    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:38:20.136199 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:20.136176    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xwkdt" event={"ID":"55d54b63-00e7-4ef2-b36b-0cc61d1a434c","Type":"ContainerStarted","Data":"91251b248118710c94851c3ddf466c11e529d6a6e0fdcace70a02b70effe851e"}
Jul 03 20:38:20.152540 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:20.152496    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qf5hm" podStartSLOduration=66.556276057 podStartE2EDuration="1m11.1524825s" totalImagesPullingTime="4.596206443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:38:20.152148761 +0000 UTC m=+72.232890170" watchObservedRunningTime="2026-07-03 20:38:20.1524825 +0000 UTC m=+72.233223888"
Jul 03 20:38:20.167871 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:20.167826    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xwkdt" podStartSLOduration=66.250740574 podStartE2EDuration="1m11.167812958s" totalImagesPullingTime="4.917072384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:38:20.167259999 +0000 UTC m=+72.248001406" watchObservedRunningTime="2026-07-03 20:38:20.167812958 +0000 UTC m=+72.248554365"
Jul 03 20:38:24.108643 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:24.108610    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:24.166879 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:24.166838    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:47.176173 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:47.176056    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:47.225908 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:47.225870    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:38:48.205643 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:48.205607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:38:48.206023 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:48.205720    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:38:48.206023 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:38:48.205779    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:39:52.205765645 +0000 UTC m=+164.286507032 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:38:51.142306 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:38:51.142275    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qf5hm"
Jul 03 20:39:18.213132 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:39:18.213077    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:39:18.213622 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:39:18.213227    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 03 20:39:18.213622 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:39:18.213286    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:41:20.213269295 +0000 UTC m=+252.294010685 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : secret "metrics-daemon-secret" not found
Jul 03 20:39:47.314852 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:39:47.314808    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pz4zv" podUID="69053652-4d62-46a7-b20e-7a963f70b45d"
Jul 03 20:39:47.379776 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:39:47.379749    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:39:47.877569 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:39:47.877529    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:39:52.252011 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:39:52.251962    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:39:52.252415 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:39:52.252113    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 03 20:39:52.252415 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:39:52.252181    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls podName:69053652-4d62-46a7-b20e-7a963f70b45d nodeName:}" failed. No retries permitted until 2026-07-03 20:41:54.252164931 +0000 UTC m=+286.332906321 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls") pod "dns-default-pz4zv" (UID: "69053652-4d62-46a7-b20e-7a963f70b45d") : secret "dns-default-metrics-tls" not found
Jul 03 20:40:01.418965 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:01.418925    2553 generic.go:356] "Generic (PLEG): container finished" podID="d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3" containerID="bc7a0eac0afc42c843d40ce0f221c048fc70437065a1bfa5ef1a711039f8e99d" exitCode=1
Jul 03 20:40:01.419358 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:01.418997    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" event={"ID":"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3","Type":"ContainerDied","Data":"bc7a0eac0afc42c843d40ce0f221c048fc70437065a1bfa5ef1a711039f8e99d"}
Jul 03 20:40:01.419358 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:01.419286    2553 scope.go:117] "RemoveContainer" containerID="bc7a0eac0afc42c843d40ce0f221c048fc70437065a1bfa5ef1a711039f8e99d"
Jul 03 20:40:01.865107 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:01.865068    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:40:02.081059 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:02.081011    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:40:02.424039 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:02.424002    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" event={"ID":"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3","Type":"ContainerStarted","Data":"f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a"}
Jul 03 20:40:02.424425 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:02.424179    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:40:02.424738 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:02.424723    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:40:53.183383 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:40:53.183347    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:16.672534 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:16.672500    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:16.775352 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:16.775318    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:19.187476 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:19.187448    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:19.237494 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:19.237462    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:20.235076 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.235034    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:41:20.235468 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:20.235204    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 03 20:41:20.235468 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:20.235276    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs podName:23f92759-61cf-4a1d-8b4a-c5304c4c0c59 nodeName:}" failed. No retries permitted until 2026-07-03 20:43:22.235259084 +0000 UTC m=+374.316000470 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs") pod "network-metrics-daemon-ts9j8" (UID: "23f92759-61cf-4a1d-8b4a-c5304c4c0c59") : secret "metrics-daemon-secret" not found
Jul 03 20:41:20.637970 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.637935    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vpgmp"]
Jul 03 20:41:20.640905 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.640865    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.644161 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.644133    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 03 20:41:20.644161 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.644133    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 03 20:41:20.644417 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.644404    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 03 20:41:20.645403 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.645390    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 03 20:41:20.645464 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.645399    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gz87q\""
Jul 03 20:41:20.660058 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.660033    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vpgmp"]
Jul 03 20:41:20.691913 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.691873    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:20.738095 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.738064    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b39e2730-3085-4abf-a0e4-4f0c308a3deb-crio-socket\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.738095 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.738098    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b39e2730-3085-4abf-a0e4-4f0c308a3deb-data-volume\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.738240 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.738114    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6grd\" (UniqueName: \"kubernetes.io/projected/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-api-access-m6grd\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.738240 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.738193    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.738240 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.738214    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.743440 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.743420    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:20.838905 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.838866    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.839043 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.838968    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b39e2730-3085-4abf-a0e4-4f0c308a3deb-crio-socket\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.839043 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.839005    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b39e2730-3085-4abf-a0e4-4f0c308a3deb-data-volume\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.839043 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:20.839011    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:20.839043 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.839029    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6grd\" (UniqueName: \"kubernetes.io/projected/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-api-access-m6grd\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.839265 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.839092    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b39e2730-3085-4abf-a0e4-4f0c308a3deb-crio-socket\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.839265 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.839092    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.839265 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:20.839150    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 03 20:41:20.839265 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:20.839172    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:21.339136281 +0000 UTC m=+253.419877676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:20.839265 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:20.839230    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:21.33921292 +0000 UTC m=+253.419954329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : configmap "kube-rbac-proxy" not found
Jul 03 20:41:20.839430 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.839354    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b39e2730-3085-4abf-a0e4-4f0c308a3deb-data-volume\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:20.847621 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:20.847598    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6grd\" (UniqueName: \"kubernetes.io/projected/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-api-access-m6grd\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:21.344165 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.344135    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:21.344165 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.344170    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:21.344579 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:21.344267    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 03 20:41:21.344579 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:21.344353    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:22.344333823 +0000 UTC m=+254.425075227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : configmap "kube-rbac-proxy" not found
Jul 03 20:41:21.344579 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:21.344360    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:21.344579 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:21.344406    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:22.344394223 +0000 UTC m=+254.425135622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:21.407750 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.407712    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:21.496607 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.496571    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:21.597139 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.597053    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:21.751360 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.751330    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:21.807061 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:21.807032    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:22.189254 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.189226    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:22.238633 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.238601    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:22.351907 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.351861    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:22.352300 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.351931    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:22.352300 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:22.351994    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 03 20:41:22.352300 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:22.352054    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:24.35203801 +0000 UTC m=+256.432779397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : configmap "kube-rbac-proxy" not found
Jul 03 20:41:22.352300 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:22.352120    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:22.352300 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:22.352194    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:24.352177072 +0000 UTC m=+256.432918465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:22.460714 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.460642    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:22.563425 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.563393    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:22.797025 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:22.797000    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpjjr_c84e01a7-e24c-4313-9fb4-636dc71bc1b2/dns-node-resolver/0.log"
Jul 03 20:41:23.796745 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:23.796718    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rcmn4_7e751430-2992-4aeb-b4c3-15660c5f3ffa/node-ca/0.log"
Jul 03 20:41:23.971527 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:23.971495    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:24.063479 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:24.063403    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:24.113657 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:24.113626    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:24.196433 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:24.196405    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6nz9p_becd33cb-180b-42fe-bccb-161a55b71620/serve-healthcheck-canary/0.log"
Jul 03 20:41:24.364998 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:24.364916    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:24.364998 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:24.364949    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:24.365183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:24.365021    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 03 20:41:24.365183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:24.365081    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:24.365183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:24.365095    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:28.365080694 +0000 UTC m=+260.445822081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : configmap "kube-rbac-proxy" not found
Jul 03 20:41:24.365183 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:24.365123    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:28.365111958 +0000 UTC m=+260.445853343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:24.899941 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:24.899905    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:27.064704 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:27.064670    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:27.605730 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:27.605697    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:27.914438 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:27.914359    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:28.014999 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.014972    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:28.115208 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.115177    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:28.215629 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.215554    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:28.271020 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.270990    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:28.326692 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.326666    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:28.396052 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.396028    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:28.396123 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:28.396057    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:28.396160 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:28.396143    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 03 20:41:28.396196 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:28.396166    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:28.396228 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:28.396206    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:36.396193369 +0000 UTC m=+268.476934755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:28.396228 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:28.396222    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:36.39621435 +0000 UTC m=+268.476955737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : configmap "kube-rbac-proxy" not found
Jul 03 20:41:29.307761 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:29.307725    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:29.705207 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:29.705118    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:29.900132 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:29.900084    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:30.103840 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:30.103766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:30.306261 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:30.306226    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:31.103831 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:31.103796    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:31.303639 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:31.303606    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:31.392282 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:31.392188    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:31.501655 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:31.501609    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:31.702464 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:31.702378    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:31.905463 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:31.905431    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:32.103417 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:32.103382    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:32.508018 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:32.507929    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:33.504755 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:33.504727    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:33.877108 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:33.877080    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:33.929869 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:33.929846    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:34.011862 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:34.011831    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:34.112693 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:34.112664    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:34.504017 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:34.503982    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:35.102029 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:35.101996    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:35.703734 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:35.703696    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:35.901520 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:35.901484    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:36.308135 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:36.308088    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:36.459346 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:36.459313    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:36.459346 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:36.459348    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:36.459579 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:36.459444    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:36.459579 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:36.459507    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls podName:b39e2730-3085-4abf-a0e4-4f0c308a3deb nodeName:}" failed. No retries permitted until 2026-07-03 20:41:52.459489821 +0000 UTC m=+284.540231210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vpgmp" (UID: "b39e2730-3085-4abf-a0e4-4f0c308a3deb") : secret "insights-runtime-extractor-tls" not found
Jul 03 20:41:36.459873 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:36.459851    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b39e2730-3085-4abf-a0e4-4f0c308a3deb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:36.504071 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:36.504041    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:36.704361 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:36.704291    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:37.705814 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:37.705784    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:37.905491 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:37.905459    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:38.100272 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:38.100231    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:38.304737 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:38.304707    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:41.118206 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:41.118173    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:41.203955 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:41.203919    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:41.389504 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:41.389426    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:41.765850 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:41.765767    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:41.819212 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:41.819182    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:41.967602 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:41.967566    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:42.014992 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:42.014964    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:42.369276 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:42.369240    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:42.417425 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:42.417398    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:42.979125 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:42.979092    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:43.179384 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:43.179355    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:43.610717 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:43.610688    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:43.795294 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:43.795260    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:43.842517 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:43.842493    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:44.678538 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:44.678506    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:44.777533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:44.777502    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:47.709409 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.709370    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:47.722191 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.722166    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d9d77fc47-7m4sv"]
Jul 03 20:41:47.724885 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.724867    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.727973 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.727917    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 03 20:41:47.727973 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.727958    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 03 20:41:47.727973 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.727974    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dxprn\""
Jul 03 20:41:47.729474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.729457    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 03 20:41:47.733817 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.733799    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 03 20:41:47.737700 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.737681    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d9d77fc47-7m4sv"]
Jul 03 20:41:47.790851 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.790827    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:47.846743 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.846711    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9850e83-282e-40b9-aee8-b26092d7be38-image-registry-private-configuration\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.846921 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.846768    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-bound-sa-token\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.846921 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.846789    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9850e83-282e-40b9-aee8-b26092d7be38-installation-pull-secrets\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.846921 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.846871    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9850e83-282e-40b9-aee8-b26092d7be38-registry-certificates\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.847093 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.846927    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9850e83-282e-40b9-aee8-b26092d7be38-trusted-ca\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.847093 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.846973    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-registry-tls\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.847093 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.847003    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9850e83-282e-40b9-aee8-b26092d7be38-ca-trust-extracted\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.847093 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.847036    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjkx7\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-kube-api-access-kjkx7\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.947851 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.947821    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9850e83-282e-40b9-aee8-b26092d7be38-image-registry-private-configuration\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.947973 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.947874    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-bound-sa-token\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.947973 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.947920    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9850e83-282e-40b9-aee8-b26092d7be38-installation-pull-secrets\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948051 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.947971    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9850e83-282e-40b9-aee8-b26092d7be38-registry-certificates\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948051 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.947997    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9850e83-282e-40b9-aee8-b26092d7be38-trusted-ca\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948051 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.948026    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-registry-tls\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948203 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.948139    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9850e83-282e-40b9-aee8-b26092d7be38-ca-trust-extracted\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948248 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.948200    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjkx7\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-kube-api-access-kjkx7\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948520 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.948494    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e9850e83-282e-40b9-aee8-b26092d7be38-ca-trust-extracted\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.948839 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.948821    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e9850e83-282e-40b9-aee8-b26092d7be38-registry-certificates\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.949160 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.949141    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9850e83-282e-40b9-aee8-b26092d7be38-trusted-ca\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.950613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.950586    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-registry-tls\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.950719 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.950699    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e9850e83-282e-40b9-aee8-b26092d7be38-installation-pull-secrets\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.950770 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.950726    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e9850e83-282e-40b9-aee8-b26092d7be38-image-registry-private-configuration\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.956038 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.956019    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-bound-sa-token\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:47.956444 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:47.956419    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjkx7\" (UniqueName: \"kubernetes.io/projected/e9850e83-282e-40b9-aee8-b26092d7be38-kube-api-access-kjkx7\") pod \"image-registry-5d9d77fc47-7m4sv\" (UID: \"e9850e83-282e-40b9-aee8-b26092d7be38\") " pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:48.034491 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:48.034468    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:48.159451 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:48.159421    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d9d77fc47-7m4sv"]
Jul 03 20:41:48.162473 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:41:48.162443    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9850e83_282e_40b9_aee8_b26092d7be38.slice/crio-88cbb7c1f1167c107bcaa743918a17f31b289c865a8418f7335e7d9a614276b5 WatchSource:0}: Error finding container 88cbb7c1f1167c107bcaa743918a17f31b289c865a8418f7335e7d9a614276b5: Status 404 returned error can't find the container with id 88cbb7c1f1167c107bcaa743918a17f31b289c865a8418f7335e7d9a614276b5
Jul 03 20:41:48.722500 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:48.722453    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv" event={"ID":"e9850e83-282e-40b9-aee8-b26092d7be38","Type":"ContainerStarted","Data":"57fbe8a63c99609aea8ee0da4c26118a039c55638edb998ae237410273aeffd8"}
Jul 03 20:41:48.722928 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:48.722523    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv" event={"ID":"e9850e83-282e-40b9-aee8-b26092d7be38","Type":"ContainerStarted","Data":"88cbb7c1f1167c107bcaa743918a17f31b289c865a8418f7335e7d9a614276b5"}
Jul 03 20:41:48.722928 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:48.722586    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:41:48.742805 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:48.742762    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv" podStartSLOduration=1.742748127 podStartE2EDuration="1.742748127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:41:48.740841286 +0000 UTC m=+280.821582693" watchObservedRunningTime="2026-07-03 20:41:48.742748127 +0000 UTC m=+280.823489535"
Jul 03 20:41:50.380715 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:41:50.380669    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pz4zv" podUID="69053652-4d62-46a7-b20e-7a963f70b45d"
Jul 03 20:41:50.728192 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:50.728118    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:41:50.795048 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:50.795012    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:50.851345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:50.851303    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:50.956640 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:50.956588    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:51.321706 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:51.321671    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:52.485340 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:52.485288    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:52.487570 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:52.487551    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b39e2730-3085-4abf-a0e4-4f0c308a3deb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vpgmp\" (UID: \"b39e2730-3085-4abf-a0e4-4f0c308a3deb\") " pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:52.749272 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:52.749188    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vpgmp"
Jul 03 20:41:52.874590 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:52.874419    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vpgmp"]
Jul 03 20:41:52.876946 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:41:52.876923    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39e2730_3085_4abf_a0e4_4f0c308a3deb.slice/crio-13b236478a8b77cf13470963bbf4e037f72ecde959b0c1e30d159c966fe1ea03 WatchSource:0}: Error finding container 13b236478a8b77cf13470963bbf4e037f72ecde959b0c1e30d159c966fe1ea03: Status 404 returned error can't find the container with id 13b236478a8b77cf13470963bbf4e037f72ecde959b0c1e30d159c966fe1ea03
Jul 03 20:41:53.740383 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:53.740303    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vpgmp" event={"ID":"b39e2730-3085-4abf-a0e4-4f0c308a3deb","Type":"ContainerStarted","Data":"13b236478a8b77cf13470963bbf4e037f72ecde959b0c1e30d159c966fe1ea03"}
Jul 03 20:41:54.301924 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:54.301873    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:41:54.304139 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:54.304110    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69053652-4d62-46a7-b20e-7a963f70b45d-metrics-tls\") pod \"dns-default-pz4zv\" (UID: \"69053652-4d62-46a7-b20e-7a963f70b45d\") " pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:41:54.331256 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:54.331232    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v2wzp\""
Jul 03 20:41:54.339580 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:54.339553    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:41:54.977451 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:54.977420    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:55.042102 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:55.042055    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" podUID="407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 03 20:41:55.043618 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:55.043574    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:55.301993 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:55.301953    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:56.571680 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:56.569318    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pz4zv"]
Jul 03 20:41:56.574574 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:41:56.574547    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69053652_4d62_46a7_b20e_7a963f70b45d.slice/crio-fdb6a8a9152508d11058b8abfe06369016f188e515cd9b1c7d5d4cf02a0abfe3 WatchSource:0}: Error finding container fdb6a8a9152508d11058b8abfe06369016f188e515cd9b1c7d5d4cf02a0abfe3: Status 404 returned error can't find the container with id fdb6a8a9152508d11058b8abfe06369016f188e515cd9b1c7d5d4cf02a0abfe3
Jul 03 20:41:56.752489 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:56.752394    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vpgmp" event={"ID":"b39e2730-3085-4abf-a0e4-4f0c308a3deb","Type":"ContainerStarted","Data":"90f1da13dfab9be2ed21b902881f55cb7eb5eced0a2c7c1ecbd11c0382598d04"}
Jul 03 20:41:56.753553 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:56.753526    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz4zv" event={"ID":"69053652-4d62-46a7-b20e-7a963f70b45d","Type":"ContainerStarted","Data":"fdb6a8a9152508d11058b8abfe06369016f188e515cd9b1c7d5d4cf02a0abfe3"}
Jul 03 20:41:57.759089 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:57.759043    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vpgmp" event={"ID":"b39e2730-3085-4abf-a0e4-4f0c308a3deb","Type":"ContainerStarted","Data":"230882bb112c02bc8604fbe32482267ca14ff9c9f29c4aa56cb8cf2617423886"}
Jul 03 20:41:58.764641 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:58.764600    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz4zv" event={"ID":"69053652-4d62-46a7-b20e-7a963f70b45d","Type":"ContainerStarted","Data":"2a760e00f84b2dce7e783b6a99426bc97ddef76f087fa6a0ef049a30ca03f05a"}
Jul 03 20:41:58.765126 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:58.764671    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz4zv" event={"ID":"69053652-4d62-46a7-b20e-7a963f70b45d","Type":"ContainerStarted","Data":"ef4ff77d93b2d47b21fa532d28735aacd50a5f94ae059fb2977b4e20a28113a1"}
Jul 03 20:41:58.765126 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:58.764823    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:41:58.781704 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:58.781612    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pz4zv" podStartSLOduration=253.183782084 podStartE2EDuration="4m14.781593897s" totalImagesPullingTime="1.597811813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:41:58.781153614 +0000 UTC m=+290.861895023" watchObservedRunningTime="2026-07-03 20:41:58.781593897 +0000 UTC m=+290.862335305"
Jul 03 20:41:58.906765 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:58.906716    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:59.396260 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:59.396225    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:59.497317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:59.497287    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:41:59.769706 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:59.769671    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vpgmp" event={"ID":"b39e2730-3085-4abf-a0e4-4f0c308a3deb","Type":"ContainerStarted","Data":"540595cc60184002fff9bc260336269e9f7a98895e73f805e3017dcf4355b01e"}
Jul 03 20:41:59.787417 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:41:59.787374    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vpgmp" podStartSLOduration=33.10198382 podStartE2EDuration="39.787358428s" totalImagesPullingTime="6.685374608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 20:41:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:41:59.785325988 +0000 UTC m=+291.866067395" watchObservedRunningTime="2026-07-03 20:41:59.787358428 +0000 UTC m=+291.868099835"
Jul 03 20:42:01.095829 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:01.095796    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:01.381956 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:01.381846    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:01.777613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:01.777584    2553 generic.go:356] "Generic (PLEG): container finished" podID="d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3" containerID="f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a" exitCode=1
Jul 03 20:42:01.777768 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:01.777657    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" event={"ID":"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3","Type":"ContainerDied","Data":"f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a"}
Jul 03 20:42:01.777768 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:01.777719    2553 scope.go:117] "RemoveContainer" containerID="bc7a0eac0afc42c843d40ce0f221c048fc70437065a1bfa5ef1a711039f8e99d"
Jul 03 20:42:01.778094 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:01.778078    2553 scope.go:117] "RemoveContainer" containerID="f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a"
Jul 03 20:42:01.778259 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:01.778228    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-585dc499df-jh4dq_open-cluster-management-agent-addon(d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" podUID="d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3"
Jul 03 20:42:02.199249 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:02.199169    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:02.299446 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:02.299407    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:02.425045 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:02.425009    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:42:02.781676 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:02.781652    2553 scope.go:117] "RemoveContainer" containerID="f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a"
Jul 03 20:42:02.781844 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:02.781803    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-585dc499df-jh4dq_open-cluster-management-agent-addon(d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" podUID="d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3"
Jul 03 20:42:04.865655 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:04.865615    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ts9j8" podUID="23f92759-61cf-4a1d-8b4a-c5304c4c0c59"
Jul 03 20:42:05.020088 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:05.020058    2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:42:05.020407 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:05.020395    2553 scope.go:117] "RemoveContainer" containerID="f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a"
Jul 03 20:42:05.020557 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:05.020538    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-585dc499df-jh4dq_open-cluster-management-agent-addon(d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" podUID="d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3"
Jul 03 20:42:05.041774 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:05.041741    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" podUID="407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 03 20:42:06.241019 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:06.240984    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:06.342129 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:06.342093    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:06.811608 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:06.811575    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:06.872346 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:06.872311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:08.039176 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:08.039142    2553 patch_prober.go:28] interesting pod/image-registry-5d9d77fc47-7m4sv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 03 20:42:08.039531 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:08.039202    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv" podUID="e9850e83-282e-40b9-aee8-b26092d7be38" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 03 20:42:08.286684 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:08.286655    2553 kubelet.go:1648] "Image garbage collection succeeded"
Jul 03 20:42:08.318431 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:08.318358    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:08.369721 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:08.369696    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:08.771677 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:08.771598    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pz4zv"
Jul 03 20:42:09.685345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.685312    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:09.729783 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.729755    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"]
Jul 03 20:42:09.732721 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.732700    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d9d77fc47-7m4sv"
Jul 03 20:42:09.732832 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.732794    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.735382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.735355    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 03 20:42:09.735488 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.735437    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 03 20:42:09.735488 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.735457    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pk45x\""
Jul 03 20:42:09.736879 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.736862    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 03 20:42:09.737006 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.736862    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 03 20:42:09.737006 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.736935    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 03 20:42:09.737006 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.736964    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 03 20:42:09.742613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.742589    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"]
Jul 03 20:42:09.759036 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.759008    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:09.809650 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.809618    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.809767 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.809654    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t7lt\" (UniqueName: \"kubernetes.io/projected/e29d6023-125a-455c-8443-af0ccc5d6453-kube-api-access-4t7lt\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.809824 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.809762    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.809883 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.809840    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e29d6023-125a-455c-8443-af0ccc5d6453-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.809883 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.809865    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.810019 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.809918    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29d6023-125a-455c-8443-af0ccc5d6453-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.878317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.878285    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:09.911137 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911102    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8qdsm"]
Jul 03 20:42:09.911137 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911155    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t7lt\" (UniqueName: \"kubernetes.io/projected/e29d6023-125a-455c-8443-af0ccc5d6453-kube-api-access-4t7lt\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911189    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911221    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e29d6023-125a-455c-8443-af0ccc5d6453-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911242    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911267    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29d6023-125a-455c-8443-af0ccc5d6453-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911703 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:09.911382    2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 03 20:42:09.911703 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:09.911449    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls podName:e29d6023-125a-455c-8443-af0ccc5d6453 nodeName:}" failed. No retries permitted until 2026-07-03 20:42:10.411431091 +0000 UTC m=+302.492172491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-6nkjq" (UID: "e29d6023-125a-455c-8443-af0ccc5d6453") : secret "kube-state-metrics-tls" not found
Jul 03 20:42:09.911703 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911653    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e29d6023-125a-455c-8443-af0ccc5d6453-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.911943 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911925    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29d6023-125a-455c-8443-af0ccc5d6453-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.912016 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.911994    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.913680 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.913662    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.914640 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.914626    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:09.917092 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.917058    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 03 20:42:09.917213 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.917093    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-55fjf\""
Jul 03 20:42:09.917213 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.917168    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 03 20:42:09.917331 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.917318    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 03 20:42:09.919861 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.919839    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t7lt\" (UniqueName: \"kubernetes.io/projected/e29d6023-125a-455c-8443-af0ccc5d6453-kube-api-access-4t7lt\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:09.937852 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:09.937801    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.012036 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012002    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-wtmp\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012063    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5mj\" (UniqueName: \"kubernetes.io/projected/dc2eec07-d321-4093-a18f-a2198810cc59-kube-api-access-5m5mj\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012083    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-root\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012162    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-sys\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012204    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-tls\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012246    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2eec07-d321-4093-a18f-a2198810cc59-metrics-client-ca\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012451 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012286    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012451 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012326    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-textfile\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.012451 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.012351    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.083405 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.083372    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.113022 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.112990    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5mj\" (UniqueName: \"kubernetes.io/projected/dc2eec07-d321-4093-a18f-a2198810cc59-kube-api-access-5m5mj\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113174 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113032    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-root\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113174 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113077    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-sys\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113174 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113108    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-tls\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113174 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113151    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2eec07-d321-4093-a18f-a2198810cc59-metrics-client-ca\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113169    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-root\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113178    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113189    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-sys\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113329    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-textfile\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113345 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:10.113339    2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 03 20:42:10.113533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113366    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113533 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:10.113402    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-tls podName:dc2eec07-d321-4093-a18f-a2198810cc59 nodeName:}" failed. No retries permitted until 2026-07-03 20:42:10.613380602 +0000 UTC m=+302.694122005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-tls") pod "node-exporter-8qdsm" (UID: "dc2eec07-d321-4093-a18f-a2198810cc59") : secret "node-exporter-tls" not found
Jul 03 20:42:10.113533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113454    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-wtmp\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113687 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113641    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-wtmp\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113736 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113719    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-textfile\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.113799 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.113778    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc2eec07-d321-4093-a18f-a2198810cc59-metrics-client-ca\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.114068 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.114051    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.115496 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.115478    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.120804 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.120780    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5mj\" (UniqueName: \"kubernetes.io/projected/dc2eec07-d321-4093-a18f-a2198810cc59-kube-api-access-5m5mj\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.184826 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.184794    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.242958 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.242857    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.415244 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.415208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:10.415404 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:10.415355    2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 03 20:42:10.415445 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:42:10.415421    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls podName:e29d6023-125a-455c-8443-af0ccc5d6453 nodeName:}" failed. No retries permitted until 2026-07-03 20:42:11.415405993 +0000 UTC m=+303.496147379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-6nkjq" (UID: "e29d6023-125a-455c-8443-af0ccc5d6453") : secret "kube-state-metrics-tls" not found
Jul 03 20:42:10.486875 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.486845    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.589265 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.589233    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.617392 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.617361    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-tls\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.619618 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.619598    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc2eec07-d321-4093-a18f-a2198810cc59-node-exporter-tls\") pod \"node-exporter-8qdsm\" (UID: \"dc2eec07-d321-4093-a18f-a2198810cc59\") " pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.687634 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.687605    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.746810 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.746780    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.807446 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.807414    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:10.831500 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.831473    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qdsm"
Jul 03 20:42:10.842677 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:42:10.842655    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2eec07_d321_4093_a18f_a2198810cc59.slice/crio-99b7ef695a39c10feb3aeff6b830176494fe3fdd4dcf82c3d92768f53b88cd44 WatchSource:0}: Error finding container 99b7ef695a39c10feb3aeff6b830176494fe3fdd4dcf82c3d92768f53b88cd44: Status 404 returned error can't find the container with id 99b7ef695a39c10feb3aeff6b830176494fe3fdd4dcf82c3d92768f53b88cd44
Jul 03 20:42:10.844309 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.844296    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 03 20:42:10.906650 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:10.906617    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:11.016279 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.016247    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:11.069060 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.069031    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:11.424483 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.424446    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:11.426962 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.426925    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e29d6023-125a-455c-8443-af0ccc5d6453-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6nkjq\" (UID: \"e29d6023-125a-455c-8443-af0ccc5d6453\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:11.539435 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.539398    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:11.541320 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.541304    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"
Jul 03 20:42:11.640711 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.640681    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:11.683346 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.683274    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq"]
Jul 03 20:42:11.692833 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:42:11.692800    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29d6023_125a_455c_8443_af0ccc5d6453.slice/crio-b54263651671e8d5c0c36d68cde1c8676234cd7e78761ff34256716330d6920e WatchSource:0}: Error finding container b54263651671e8d5c0c36d68cde1c8676234cd7e78761ff34256716330d6920e: Status 404 returned error can't find the container with id b54263651671e8d5c0c36d68cde1c8676234cd7e78761ff34256716330d6920e
Jul 03 20:42:11.810195 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.810153    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qdsm" event={"ID":"dc2eec07-d321-4093-a18f-a2198810cc59","Type":"ContainerStarted","Data":"99b7ef695a39c10feb3aeff6b830176494fe3fdd4dcf82c3d92768f53b88cd44"}
Jul 03 20:42:11.811325 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:11.811294    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq" event={"ID":"e29d6023-125a-455c-8443-af0ccc5d6453","Type":"ContainerStarted","Data":"b54263651671e8d5c0c36d68cde1c8676234cd7e78761ff34256716330d6920e"}
Jul 03 20:42:12.734055 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:12.734026    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:12.817292 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:12.817251    2553 generic.go:356] "Generic (PLEG): container finished" podID="dc2eec07-d321-4093-a18f-a2198810cc59" containerID="7e5bbb54203488813054ad32efd94df176945cfbcced3e49ae14d4f91e29ab98" exitCode=0
Jul 03 20:42:12.817469 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:12.817300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qdsm" event={"ID":"dc2eec07-d321-4093-a18f-a2198810cc59","Type":"ContainerDied","Data":"7e5bbb54203488813054ad32efd94df176945cfbcced3e49ae14d4f91e29ab98"}
Jul 03 20:42:12.831431 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:12.831397    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:12.932128 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:12.932106    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:13.822263 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.822231    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq" event={"ID":"e29d6023-125a-455c-8443-af0ccc5d6453","Type":"ContainerStarted","Data":"cdd49d3ef184e8d4d5e20b96e9c57de75ac5659a42b93e4c1855d2993e4c9d3d"}
Jul 03 20:42:13.822675 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.822293    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq" event={"ID":"e29d6023-125a-455c-8443-af0ccc5d6453","Type":"ContainerStarted","Data":"a1f0caaa5561b5b1760247da96176d31238c7ec81d692ef7ffc9c778cc6c3c9f"}
Jul 03 20:42:13.822675 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.822320    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq" event={"ID":"e29d6023-125a-455c-8443-af0ccc5d6453","Type":"ContainerStarted","Data":"b9f25d04543687fd92bbc31c06e1fd8d91f5a5b3a74f2e2c8646b50b889afc3c"}
Jul 03 20:42:13.824350 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.824325    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qdsm" event={"ID":"dc2eec07-d321-4093-a18f-a2198810cc59","Type":"ContainerStarted","Data":"805f85466c497feb9f7e38756c390d01d5d2bb60fad49e5504800b717bd86e42"}
Jul 03 20:42:13.824460 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.824364    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qdsm" event={"ID":"dc2eec07-d321-4093-a18f-a2198810cc59","Type":"ContainerStarted","Data":"61421df718d79a034ab6e532d3cd390e72c2f9bf4dcaa3edd173b8cc2faaff2e"}
Jul 03 20:42:13.838808 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.838768    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6nkjq" podStartSLOduration=3.629659661 podStartE2EDuration="4.838753964s" totalImagesPullingTime="1.209094303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:42:13.83817713 +0000 UTC m=+305.918918538" watchObservedRunningTime="2026-07-03 20:42:13.838753964 +0000 UTC m=+305.919495373"
Jul 03 20:42:13.855108 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:13.855063    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8qdsm" podStartSLOduration=3.6050480240000002 podStartE2EDuration="4.855050501s" totalImagesPullingTime="1.186507332s" totalInitContainerRuntime="63.495145ms" isStatefulPod=true podCreationTimestamp="2026-07-03 20:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:42:13.854278021 +0000 UTC m=+305.935019440" watchObservedRunningTime="2026-07-03 20:42:13.855050501 +0000 UTC m=+305.935791908"
Jul 03 20:42:14.496315 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:14.496278    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:14.623441 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:14.623405    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:14.700363 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:14.700322    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:14.817346 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:14.817316    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:15.042424 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.042383    2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" podUID="407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 03 20:42:15.042751 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.042453    2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl"
Jul 03 20:42:15.042958 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.042926    2553 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"cd00c218ab8ecf7452e2db3e903a0f12d35f25796126eb8e3463497019576bfd"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 03 20:42:15.042997 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.042980    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" podUID="407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5" containerName="service-proxy" containerID="cri-o://cd00c218ab8ecf7452e2db3e903a0f12d35f25796126eb8e3463497019576bfd" gracePeriod=30
Jul 03 20:42:15.833835 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.833802    2553 generic.go:356] "Generic (PLEG): container finished" podID="407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5" containerID="cd00c218ab8ecf7452e2db3e903a0f12d35f25796126eb8e3463497019576bfd" exitCode=2
Jul 03 20:42:15.834046 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.833874    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" event={"ID":"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5","Type":"ContainerDied","Data":"cd00c218ab8ecf7452e2db3e903a0f12d35f25796126eb8e3463497019576bfd"}
Jul 03 20:42:15.834046 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:15.833947    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-548dfc4744-4w9fl" event={"ID":"407f9e13-1ebd-4f81-a4b7-0e3b0e73b0f5","Type":"ContainerStarted","Data":"85aeee74ef634c1c57ba3dad1663142a0e2009a9e6a836709801692cdce3023e"}
Jul 03 20:42:16.011183 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:16.011149    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:17.601200 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:17.601168    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:17.703823 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:17.703788    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:17.864511 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:17.864442    2553 scope.go:117] "RemoveContainer" containerID="f7c8dd82191bf0c653579634570f4d9081a6d6291b2ce946aa9e6e3a8fff859a"
Jul 03 20:42:17.895951 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:17.895928    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:17.999251 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:17.999218    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:18.844631 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:18.844597    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq" event={"ID":"d2736f21-ac43-42e5-b3c1-2aeb54f5c3d3","Type":"ContainerStarted","Data":"06ec641cae628b294d9d89a609e435a94adbbb89ac4b326ae82423553fc324e9"}
Jul 03 20:42:18.845140 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:18.844903    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:42:18.846577 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:18.846558    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-585dc499df-jh4dq"
Jul 03 20:42:18.864260 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:18.864232    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:42:20.099582 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:20.099550    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:20.150484 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:20.150454    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:26.705541 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:26.705507    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:26.754687 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:26.754660    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:26.899783 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:26.899756    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:26.952159 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:26.952121    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:29.685584 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:29.685557    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:29.755092 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:29.755060    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:30.370718 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:30.370672    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:30.468679 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:30.468643    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:44.684736 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:44.684703    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:42:44.739681 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:42:44.739648    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:22.267367 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:22.267322    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:43:22.269708 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:22.269686    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23f92759-61cf-4a1d-8b4a-c5304c4c0c59-metrics-certs\") pod \"network-metrics-daemon-ts9j8\" (UID: \"23f92759-61cf-4a1d-8b4a-c5304c4c0c59\") " pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:43:22.467654 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:22.467621    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4xwrm\""
Jul 03 20:43:22.475146 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:22.475124    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ts9j8"
Jul 03 20:43:22.600406 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:22.600376    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ts9j8"]
Jul 03 20:43:22.603507 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:43:22.603480    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f92759_61cf_4a1d_8b4a_c5304c4c0c59.slice/crio-820b0886f03de19d85a610764fad22c8d2e53b2aa07b3c4fbe5c2259aaee25f2 WatchSource:0}: Error finding container 820b0886f03de19d85a610764fad22c8d2e53b2aa07b3c4fbe5c2259aaee25f2: Status 404 returned error can't find the container with id 820b0886f03de19d85a610764fad22c8d2e53b2aa07b3c4fbe5c2259aaee25f2
Jul 03 20:43:23.052334 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:23.052302    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ts9j8" event={"ID":"23f92759-61cf-4a1d-8b4a-c5304c4c0c59","Type":"ContainerStarted","Data":"820b0886f03de19d85a610764fad22c8d2e53b2aa07b3c4fbe5c2259aaee25f2"}
Jul 03 20:43:23.531447 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:23.531383    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:23.631794 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:23.631761    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:24.057450 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:24.057416    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ts9j8" event={"ID":"23f92759-61cf-4a1d-8b4a-c5304c4c0c59","Type":"ContainerStarted","Data":"38759f45375b8f473f28629d4ff45c3b44834946e871b31700d4ed3872da7680"}
Jul 03 20:43:24.057596 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:24.057482    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ts9j8" event={"ID":"23f92759-61cf-4a1d-8b4a-c5304c4c0c59","Type":"ContainerStarted","Data":"2b01804042613445fcb8dd3c1348eea0a59133dbc15bbf3f7453e3af354afbef"}
Jul 03 20:43:24.074398 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:24.074344    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ts9j8" podStartSLOduration=374.11920358 podStartE2EDuration="6m15.074329314s" totalImagesPullingTime="955.125734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:43:24.073099428 +0000 UTC m=+376.153840837" watchObservedRunningTime="2026-07-03 20:43:24.074329314 +0000 UTC m=+376.155070721"
Jul 03 20:43:24.157616 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:24.157564    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:24.221147 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:24.221112    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:25.920665 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:25.920635    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:25.986974 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:25.986944    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:30.792188 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:30.792151    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:30.842376 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:30.842332    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:37.792182 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:37.792147    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:37.840047 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:37.840018    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:41.398191 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:41.398149    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:41.499915 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:41.499869    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:41.599404 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:41.599375    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:42.123858 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:42.123824    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:42.185922 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:42.185872    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:44.725327 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:44.725294    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:44.814267 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:44.814237    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:52.727400 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:52.727362    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:52.828058 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:52.828027    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:43:52.901609 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:43:52.901582    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:01.980752 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:01.980716    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:02.066705 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:02.066677    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:16.045420 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:16.045374    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:16.239473 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:16.239441    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:23.632276 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:23.632247    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:24.847203 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:24.847168    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:27.886019 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:27.885985    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:27.991880 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:27.991852    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:30.150390 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:30.150315    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:30.293927 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:30.293875    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:30.502293 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:30.502209    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:32.713415 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:32.713382    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:33.132096 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:33.132067    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:34.834875 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:34.834845    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:34.935342 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:34.935311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:38.808396 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:38.808368    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:38.909320 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:38.909283    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:39.934477 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:39.934447    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:40.837377 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:40.837332    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:47.260080 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:47.260041    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:47.351128 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:47.351091    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:48.446710 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:48.446680    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:48.499065 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:48.499036    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:49.278116 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.278082    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"]
Jul 03 20:44:49.280873 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.280858    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.283940 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.283914    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 03 20:44:49.285065 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.285044    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zmp6w\""
Jul 03 20:44:49.285173 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.285073    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 03 20:44:49.292584 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.292565    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"]
Jul 03 20:44:49.387464 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.387427    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee77cc6c-d5c9-47a2-9527-3313cd65e689-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kc5hl\" (UID: \"ee77cc6c-d5c9-47a2-9527-3313cd65e689\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.387639 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.387509    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmsl7\" (UniqueName: \"kubernetes.io/projected/ee77cc6c-d5c9-47a2-9527-3313cd65e689-kube-api-access-tmsl7\") pod \"openshift-lws-operator-fd8ccff4c-kc5hl\" (UID: \"ee77cc6c-d5c9-47a2-9527-3313cd65e689\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.488349 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.488317    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmsl7\" (UniqueName: \"kubernetes.io/projected/ee77cc6c-d5c9-47a2-9527-3313cd65e689-kube-api-access-tmsl7\") pod \"openshift-lws-operator-fd8ccff4c-kc5hl\" (UID: \"ee77cc6c-d5c9-47a2-9527-3313cd65e689\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.488349 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.488357    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee77cc6c-d5c9-47a2-9527-3313cd65e689-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kc5hl\" (UID: \"ee77cc6c-d5c9-47a2-9527-3313cd65e689\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.488775 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.488665    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee77cc6c-d5c9-47a2-9527-3313cd65e689-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kc5hl\" (UID: \"ee77cc6c-d5c9-47a2-9527-3313cd65e689\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.499334 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.499305    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmsl7\" (UniqueName: \"kubernetes.io/projected/ee77cc6c-d5c9-47a2-9527-3313cd65e689-kube-api-access-tmsl7\") pod \"openshift-lws-operator-fd8ccff4c-kc5hl\" (UID: \"ee77cc6c-d5c9-47a2-9527-3313cd65e689\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.589599 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.589512    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"
Jul 03 20:44:49.716545 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:49.716514    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl"]
Jul 03 20:44:49.719626 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:44:49.719598    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee77cc6c_d5c9_47a2_9527_3313cd65e689.slice/crio-1839eca7472ef3927f95ca3c0259c8a586c699907cbe714df23e575426ce2145 WatchSource:0}: Error finding container 1839eca7472ef3927f95ca3c0259c8a586c699907cbe714df23e575426ce2145: Status 404 returned error can't find the container with id 1839eca7472ef3927f95ca3c0259c8a586c699907cbe714df23e575426ce2145
Jul 03 20:44:50.345150 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:50.345104    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl" event={"ID":"ee77cc6c-d5c9-47a2-9527-3313cd65e689","Type":"ContainerStarted","Data":"1839eca7472ef3927f95ca3c0259c8a586c699907cbe714df23e575426ce2145"}
Jul 03 20:44:52.355271 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:52.355164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl" event={"ID":"ee77cc6c-d5c9-47a2-9527-3313cd65e689","Type":"ContainerStarted","Data":"536556b5c3e06f3ee3127bfd77e23eaee64ba2115e2d97d1719fda24deb1c0aa"}
Jul 03 20:44:52.372909 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:52.372840    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kc5hl" podStartSLOduration=0.999832642 podStartE2EDuration="3.372825634s" totalImagesPullingTime="2.372992992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:44:52.371812926 +0000 UTC m=+464.452554336" watchObservedRunningTime="2026-07-03 20:44:52.372825634 +0000 UTC m=+464.453567041"
Jul 03 20:44:52.998573 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:52.998542    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:53.099846 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:53.099798    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:44:55.750203 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:44:55.750172    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:09.677826 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:09.677788    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:09.769731 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:09.769696    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:10.316369 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.316331    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:10.570057 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.569947    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:10.678397 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.678357    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:10.708043 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.708017    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"]
Jul 03 20:45:10.715429 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.715409    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.718276 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.718254    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 03 20:45:10.718390 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.718316    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 03 20:45:10.718390 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.718316    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8vxtf\""
Jul 03 20:45:10.718500 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.718406    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 03 20:45:10.718683 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.718669    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 03 20:45:10.723621 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.723593    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"]
Jul 03 20:45:10.739972 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.739948    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-tmp\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.740065 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.739976    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.740065 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.739997    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-webhook-cert\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.740065 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.740060    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-manifests\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.740167 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.740083    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtvr\" (UniqueName: \"kubernetes.io/projected/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-kube-api-access-9xtvr\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.823674 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.823582    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:10.840929 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.840884    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-manifests\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.841014 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.840943    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtvr\" (UniqueName: \"kubernetes.io/projected/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-kube-api-access-9xtvr\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.841063 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.841013    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-tmp\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.841063 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.841039    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.841146 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.841063    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-webhook-cert\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.841417 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.841391    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-manifests\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.843260 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.843237    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-tmp\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.843467 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.843446    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-webhook-cert\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.843521 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.843450    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.849580 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.849559    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtvr\" (UniqueName: \"kubernetes.io/projected/4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155-kube-api-access-9xtvr\") pod \"opendatahub-operator-controller-manager-5448f4d7cf-vwdqt\" (UID: \"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155\") " pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:10.925109 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:10.925079    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:11.024685 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:11.024658    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:11.026593 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:11.026572    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:11.151113 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:11.151080    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:11.170746 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:11.170724    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"]
Jul 03 20:45:11.174523 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:11.174500    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed4cc2b_e67f_42f7_9ef3_c3f23caf0155.slice/crio-377c3cd75e9fccaa101279ffca5b8e20aa2a80b7fc6111b5df4bb14a6c570bc5 WatchSource:0}: Error finding container 377c3cd75e9fccaa101279ffca5b8e20aa2a80b7fc6111b5df4bb14a6c570bc5: Status 404 returned error can't find the container with id 377c3cd75e9fccaa101279ffca5b8e20aa2a80b7fc6111b5df4bb14a6c570bc5
Jul 03 20:45:11.248326 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:11.248296    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:11.421164 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:11.421087    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt" event={"ID":"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155","Type":"ContainerStarted","Data":"377c3cd75e9fccaa101279ffca5b8e20aa2a80b7fc6111b5df4bb14a6c570bc5"}
Jul 03 20:45:14.101983 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:14.101958    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:14.202040 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:14.202009    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:14.433929 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:14.433876    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt" event={"ID":"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155","Type":"ContainerStarted","Data":"623f3b72adc986129fd844fd8ab0f39f32f2aee12c0d727a08cdc30e548e7058"}
Jul 03 20:45:15.438535 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:15.438448    2553 generic.go:356] "Generic (PLEG): container finished" podID="4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155" containerID="623f3b72adc986129fd844fd8ab0f39f32f2aee12c0d727a08cdc30e548e7058" exitCode=0
Jul 03 20:45:15.438535 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:15.438499    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt" event={"ID":"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155","Type":"ContainerDied","Data":"623f3b72adc986129fd844fd8ab0f39f32f2aee12c0d727a08cdc30e548e7058"}
Jul 03 20:45:16.444808 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:16.444769    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt" event={"ID":"4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155","Type":"ContainerStarted","Data":"722436e2022190cd30d412b0cc3c053ee13b883439b8a1419abadc213482f519"}
Jul 03 20:45:16.445257 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:16.444938    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:16.468911 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:16.468843    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt" podStartSLOduration=2.263943186 podStartE2EDuration="6.468828876s" totalImagesPullingTime="3.177895542s" totalInitContainerRuntime="1.026990148s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:16.468231932 +0000 UTC m=+488.548973340" watchObservedRunningTime="2026-07-03 20:45:16.468828876 +0000 UTC m=+488.549570283"
Jul 03 20:45:24.517344 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:24.517286    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:24.580851 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:24.580795    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:27.450445 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:27.450414    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5448f4d7cf-vwdqt"
Jul 03 20:45:29.702873 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:29.702837    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:29.823744 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:29.823659    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:30.479396 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.479360    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8fsbz"]
Jul 03 20:45:30.487866 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.487839    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:30.490525 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.490503    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tpx4n\""
Jul 03 20:45:30.490648 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.490521    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 03 20:45:30.490845 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.490818    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8fsbz"]
Jul 03 20:45:30.553865 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.553811    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:30.591977 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.591950    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5lgl\" (UniqueName: \"kubernetes.io/projected/ce70dfc8-14cb-4833-912e-7c0cadd96d09-kube-api-access-r5lgl\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:30.592131 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.592008    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce70dfc8-14cb-4833-912e-7c0cadd96d09-cert\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:30.655404 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.655370    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:30.693303 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.693274    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce70dfc8-14cb-4833-912e-7c0cadd96d09-cert\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:30.693426 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.693322    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5lgl\" (UniqueName: \"kubernetes.io/projected/ce70dfc8-14cb-4833-912e-7c0cadd96d09-kube-api-access-r5lgl\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:30.693465 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:45:30.693424    2553 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 03 20:45:30.693498 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:45:30.693486    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce70dfc8-14cb-4833-912e-7c0cadd96d09-cert podName:ce70dfc8-14cb-4833-912e-7c0cadd96d09 nodeName:}" failed. No retries permitted until 2026-07-03 20:45:31.193469859 +0000 UTC m=+503.274211250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ce70dfc8-14cb-4833-912e-7c0cadd96d09-cert") pod "odh-model-controller-7998c67dc4-8fsbz" (UID: "ce70dfc8-14cb-4833-912e-7c0cadd96d09") : secret "odh-model-controller-webhook-cert" not found
Jul 03 20:45:30.702610 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.702575    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5lgl\" (UniqueName: \"kubernetes.io/projected/ce70dfc8-14cb-4833-912e-7c0cadd96d09-kube-api-access-r5lgl\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:30.759596 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:30.759514    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:31.076204 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.076166    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-92l6h"]
Jul 03 20:45:31.079825 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.079805    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.082530 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.082508    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 03 20:45:31.092237 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.092216    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:31.094288 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.094267    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-92l6h"]
Jul 03 20:45:31.196669 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.196635    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7011f91d-9e43-4583-9d1c-904fef74566e-tls-certs\") pod \"model-serving-api-54d89bd5fc-92l6h\" (UID: \"7011f91d-9e43-4583-9d1c-904fef74566e\") " pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.196836 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.196706    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssn2w\" (UniqueName: \"kubernetes.io/projected/7011f91d-9e43-4583-9d1c-904fef74566e-kube-api-access-ssn2w\") pod \"model-serving-api-54d89bd5fc-92l6h\" (UID: \"7011f91d-9e43-4583-9d1c-904fef74566e\") " pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.196836 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.196753    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce70dfc8-14cb-4833-912e-7c0cadd96d09-cert\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:31.197390 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.197353    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:31.199409 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.199380    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce70dfc8-14cb-4833-912e-7c0cadd96d09-cert\") pod \"odh-model-controller-7998c67dc4-8fsbz\" (UID: \"ce70dfc8-14cb-4833-912e-7c0cadd96d09\") " pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:31.284192 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.284153    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:31.297168 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.297146    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssn2w\" (UniqueName: \"kubernetes.io/projected/7011f91d-9e43-4583-9d1c-904fef74566e-kube-api-access-ssn2w\") pod \"model-serving-api-54d89bd5fc-92l6h\" (UID: \"7011f91d-9e43-4583-9d1c-904fef74566e\") " pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.297242 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.297218    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7011f91d-9e43-4583-9d1c-904fef74566e-tls-certs\") pod \"model-serving-api-54d89bd5fc-92l6h\" (UID: \"7011f91d-9e43-4583-9d1c-904fef74566e\") " pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.299551 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.299527    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7011f91d-9e43-4583-9d1c-904fef74566e-tls-certs\") pod \"model-serving-api-54d89bd5fc-92l6h\" (UID: \"7011f91d-9e43-4583-9d1c-904fef74566e\") " pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.305862 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.305840    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssn2w\" (UniqueName: \"kubernetes.io/projected/7011f91d-9e43-4583-9d1c-904fef74566e-kube-api-access-ssn2w\") pod \"model-serving-api-54d89bd5fc-92l6h\" (UID: \"7011f91d-9e43-4583-9d1c-904fef74566e\") " pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.392757 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.392677    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:31.400818 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.400786    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:31.537608 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.537314    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-92l6h"]
Jul 03 20:45:31.540928 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:31.540882    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7011f91d_9e43_4583_9d1c_904fef74566e.slice/crio-e1f71a6c9e0d0c8216921275befc7f26d3d65f4643d174e2ecaa908bdc9aefa2 WatchSource:0}: Error finding container e1f71a6c9e0d0c8216921275befc7f26d3d65f4643d174e2ecaa908bdc9aefa2: Status 404 returned error can't find the container with id e1f71a6c9e0d0c8216921275befc7f26d3d65f4643d174e2ecaa908bdc9aefa2
Jul 03 20:45:31.563885 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.563860    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8fsbz"]
Jul 03 20:45:31.566255 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:31.566236    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce70dfc8_14cb_4833_912e_7c0cadd96d09.slice/crio-de59c57383142c9f886748148679e67b31f730147938fd3221d0b31dac587ce3 WatchSource:0}: Error finding container de59c57383142c9f886748148679e67b31f730147938fd3221d0b31dac587ce3: Status 404 returned error can't find the container with id de59c57383142c9f886748148679e67b31f730147938fd3221d0b31dac587ce3
Jul 03 20:45:31.695709 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.695612    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:31.837564 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.837528    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"]
Jul 03 20:45:31.841050 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.841025    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:31.843441 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.843421    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 03 20:45:31.843837 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.843812    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 03 20:45:31.843951 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.843857    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 03 20:45:31.844880 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.844859    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r2b6w\""
Jul 03 20:45:31.845015 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.844867    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 03 20:45:31.849776 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.849756    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"]
Jul 03 20:45:31.902141 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.902113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f6782e-69ba-436a-b291-948e6aa42a1c-tls-certs\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:31.902307 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.902164    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f6782e-69ba-436a-b291-948e6aa42a1c-tmp\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:31.902307 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:31.902194    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6sz9\" (UniqueName: \"kubernetes.io/projected/a2f6782e-69ba-436a-b291-948e6aa42a1c-kube-api-access-t6sz9\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.003305 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.003205    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6sz9\" (UniqueName: \"kubernetes.io/projected/a2f6782e-69ba-436a-b291-948e6aa42a1c-kube-api-access-t6sz9\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.003601 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.003578    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f6782e-69ba-436a-b291-948e6aa42a1c-tls-certs\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.003706 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.003689    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f6782e-69ba-436a-b291-948e6aa42a1c-tmp\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.006007 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.005977    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2f6782e-69ba-436a-b291-948e6aa42a1c-tmp\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.006443 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.006419    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f6782e-69ba-436a-b291-948e6aa42a1c-tls-certs\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.012202 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.012176    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6sz9\" (UniqueName: \"kubernetes.io/projected/a2f6782e-69ba-436a-b291-948e6aa42a1c-kube-api-access-t6sz9\") pod \"kube-auth-proxy-57dff55469-vpmvc\" (UID: \"a2f6782e-69ba-436a-b291-948e6aa42a1c\") " pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.022614 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.022586    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:32.151565 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.151514    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"
Jul 03 20:45:32.315682 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.315611    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dff55469-vpmvc"]
Jul 03 20:45:32.319818 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:32.319787    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f6782e_69ba_436a_b291_948e6aa42a1c.slice/crio-2d28d289d0966d708ee078438e7b1e5ac4d67e8e3083a21b12dc4590f70db60a WatchSource:0}: Error finding container 2d28d289d0966d708ee078438e7b1e5ac4d67e8e3083a21b12dc4590f70db60a: Status 404 returned error can't find the container with id 2d28d289d0966d708ee078438e7b1e5ac4d67e8e3083a21b12dc4590f70db60a
Jul 03 20:45:32.514189 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.514056    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz" event={"ID":"ce70dfc8-14cb-4833-912e-7c0cadd96d09","Type":"ContainerStarted","Data":"de59c57383142c9f886748148679e67b31f730147938fd3221d0b31dac587ce3"}
Jul 03 20:45:32.519422 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.519330    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc" event={"ID":"a2f6782e-69ba-436a-b291-948e6aa42a1c","Type":"ContainerStarted","Data":"2d28d289d0966d708ee078438e7b1e5ac4d67e8e3083a21b12dc4590f70db60a"}
Jul 03 20:45:32.521393 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.521333    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h" event={"ID":"7011f91d-9e43-4583-9d1c-904fef74566e","Type":"ContainerStarted","Data":"e1f71a6c9e0d0c8216921275befc7f26d3d65f4643d174e2ecaa908bdc9aefa2"}
Jul 03 20:45:32.776611 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.776565    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"]
Jul 03 20:45:32.779483 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.779237    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.783037 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.782258    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:32.785331 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.785305    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 03 20:45:32.785596 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.785572    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 03 20:45:32.785802 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.785779    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2lkfv\""
Jul 03 20:45:32.786207 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.786001    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 03 20:45:32.786207 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.786019    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 03 20:45:32.794003 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.793982    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"]
Jul 03 20:45:32.874985 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.874915    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914265    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914315    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-local-certs\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914348    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-cacerts\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914370    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-token\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914454    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pl8\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-kube-api-access-j8pl8\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:32.914613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:32.914479    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015506    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pl8\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-kube-api-access-j8pl8\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015550    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015626    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015653    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-local-certs\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015680    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-cacerts\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015700    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.016942 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.015735    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-token\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.018647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.018008    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.031318 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.030734    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.031318 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.030755    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-cacerts\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.031318 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.031094    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.034259 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.034206    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pl8\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-kube-api-access-j8pl8\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.034259 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.034206    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-token\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.037554 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.037499    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-local-certs\") pod \"istiod-openshift-gateway-7574459656-9sbzt\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.101468 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.101138    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:33.633793 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:33.633762    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:35.876840 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:35.876813    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"]
Jul 03 20:45:35.878662 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:35.878603    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8e5d5c_ca74_4add_8cd5_3c02fa2f7446.slice/crio-aa574df0797a679c5b6ada0f37e64cfb6dede651e15eeb83b51d6dee92a8da05 WatchSource:0}: Error finding container aa574df0797a679c5b6ada0f37e64cfb6dede651e15eeb83b51d6dee92a8da05: Status 404 returned error can't find the container with id aa574df0797a679c5b6ada0f37e64cfb6dede651e15eeb83b51d6dee92a8da05
Jul 03 20:45:36.544273 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.544237    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc" event={"ID":"a2f6782e-69ba-436a-b291-948e6aa42a1c","Type":"ContainerStarted","Data":"fe5f825de18b271b22b62bf4d2c203ae7be86e0b615ea6138ba9302a2470218f"}
Jul 03 20:45:36.546052 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.546023    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h" event={"ID":"7011f91d-9e43-4583-9d1c-904fef74566e","Type":"ContainerStarted","Data":"c8049bdd1ac911228c403bbd34279a72bd7cffcbfc0a8c0a89a11eb5c531d7d4"}
Jul 03 20:45:36.546201 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.546086    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:36.547433 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.547402    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz" event={"ID":"ce70dfc8-14cb-4833-912e-7c0cadd96d09","Type":"ContainerStarted","Data":"49295cfe360f861553fedf04d032803633a42fee2754a790d1144a2a52840933"}
Jul 03 20:45:36.547548 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.547464    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:36.548613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.548592    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" event={"ID":"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446","Type":"ContainerStarted","Data":"aa574df0797a679c5b6ada0f37e64cfb6dede651e15eeb83b51d6dee92a8da05"}
Jul 03 20:45:36.562238 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.562198    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57dff55469-vpmvc" podStartSLOduration=2.166947008 podStartE2EDuration="5.562187347s" totalImagesPullingTime="3.395240339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:36.560979035 +0000 UTC m=+508.641720445" watchObservedRunningTime="2026-07-03 20:45:36.562187347 +0000 UTC m=+508.642928754"
Jul 03 20:45:36.576661 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.576615    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz" podStartSLOduration=2.426334064 podStartE2EDuration="6.576598974s" totalImagesPullingTime="4.15026491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:36.576542529 +0000 UTC m=+508.657283938" watchObservedRunningTime="2026-07-03 20:45:36.576598974 +0000 UTC m=+508.657340382"
Jul 03 20:45:36.594124 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:36.594074    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h" podStartSLOduration=1.427326122 podStartE2EDuration="5.594060993s" totalImagesPullingTime="4.166734871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:36.593210877 +0000 UTC m=+508.673952266" watchObservedRunningTime="2026-07-03 20:45:36.594060993 +0000 UTC m=+508.674802401"
Jul 03 20:45:39.836628 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:39.836583    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:45:39.837255 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:39.836674    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:45:40.566243 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.566209    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" event={"ID":"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446","Type":"ContainerStarted","Data":"b5a50cce6416da2acc06f9fa9502dcdd9564970c68a4829f7d457c828ed6c9f8"}
Jul 03 20:45:40.566531 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.566512    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:40.568017 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.567980    2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9sbzt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 03 20:45:40.568115 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.568041    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 03 20:45:40.629973 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.629944    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:40.670046 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.669989    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" podStartSLOduration=4.71432056 podStartE2EDuration="8.669968776s" totalImagesPullingTime="3.955648216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:40.667480025 +0000 UTC m=+512.748221469" watchObservedRunningTime="2026-07-03 20:45:40.669968776 +0000 UTC m=+512.750710185"
Jul 03 20:45:40.759071 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.759041    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:40.906512 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.906399    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:40.949114 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.949083    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"]
Jul 03 20:45:40.952509 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.952485    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:40.955326 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.955304    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 03 20:45:40.955538 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.955522    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-m5xx9\""
Jul 03 20:45:40.960788 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.960765    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"]
Jul 03 20:45:40.964041 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.964023    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:40.966580 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.966561    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-clxnd\""
Jul 03 20:45:40.966677 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.966589    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 03 20:45:40.966760 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.966745    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"]
Jul 03 20:45:40.978252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.978227    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"]
Jul 03 20:45:40.988150 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:40.988115    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:41.086775 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.086746    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96gbp\" (UniqueName: \"kubernetes.io/projected/8fc5fe7c-f029-48eb-89f7-280833e21ba2-kube-api-access-96gbp\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.086775 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.086777    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp254\" (UniqueName: \"kubernetes.io/projected/859207ed-804c-4ab1-bf88-2b745e381487-kube-api-access-bp254\") pod \"kserve-controller-manager-d6f4b65b8-v6cj5\" (UID: \"859207ed-804c-4ab1-bf88-2b745e381487\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.086988 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.086803    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859207ed-804c-4ab1-bf88-2b745e381487-cert\") pod \"kserve-controller-manager-d6f4b65b8-v6cj5\" (UID: \"859207ed-804c-4ab1-bf88-2b745e381487\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.086988 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.086957    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fc5fe7c-f029-48eb-89f7-280833e21ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.181191 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.181114    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:41.187749 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.187730    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fc5fe7c-f029-48eb-89f7-280833e21ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.187810 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.187770    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96gbp\" (UniqueName: \"kubernetes.io/projected/8fc5fe7c-f029-48eb-89f7-280833e21ba2-kube-api-access-96gbp\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.187810 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.187796    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp254\" (UniqueName: \"kubernetes.io/projected/859207ed-804c-4ab1-bf88-2b745e381487-kube-api-access-bp254\") pod \"kserve-controller-manager-d6f4b65b8-v6cj5\" (UID: \"859207ed-804c-4ab1-bf88-2b745e381487\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.187884 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.187829    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859207ed-804c-4ab1-bf88-2b745e381487-cert\") pod \"kserve-controller-manager-d6f4b65b8-v6cj5\" (UID: \"859207ed-804c-4ab1-bf88-2b745e381487\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.187884 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:45:41.187870    2553 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 03 20:45:41.187986 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:45:41.187944    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc5fe7c-f029-48eb-89f7-280833e21ba2-cert podName:8fc5fe7c-f029-48eb-89f7-280833e21ba2 nodeName:}" failed. No retries permitted until 2026-07-03 20:45:41.687928021 +0000 UTC m=+513.768669406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8fc5fe7c-f029-48eb-89f7-280833e21ba2-cert") pod "llmisvc-controller-manager-777c6bd7d5-bxf2b" (UID: "8fc5fe7c-f029-48eb-89f7-280833e21ba2") : secret "llmisvc-webhook-server-cert" not found
Jul 03 20:45:41.190287 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.190266    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/859207ed-804c-4ab1-bf88-2b745e381487-cert\") pod \"kserve-controller-manager-d6f4b65b8-v6cj5\" (UID: \"859207ed-804c-4ab1-bf88-2b745e381487\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.197515 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.197496    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96gbp\" (UniqueName: \"kubernetes.io/projected/8fc5fe7c-f029-48eb-89f7-280833e21ba2-kube-api-access-96gbp\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.203780 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.203755    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp254\" (UniqueName: \"kubernetes.io/projected/859207ed-804c-4ab1-bf88-2b745e381487-kube-api-access-bp254\") pod \"kserve-controller-manager-d6f4b65b8-v6cj5\" (UID: \"859207ed-804c-4ab1-bf88-2b745e381487\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.262594 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.262572    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:41.333267 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.333218    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:41.400284 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.400232    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"]
Jul 03 20:45:41.402658 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:41.402629    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859207ed_804c_4ab1_bf88_2b745e381487.slice/crio-d2b3c28b28ebff57c7c68fb1ec63f8b4d3b86292859823cefa3550d95e69c9ed WatchSource:0}: Error finding container d2b3c28b28ebff57c7c68fb1ec63f8b4d3b86292859823cefa3550d95e69c9ed: Status 404 returned error can't find the container with id d2b3c28b28ebff57c7c68fb1ec63f8b4d3b86292859823cefa3550d95e69c9ed
Jul 03 20:45:41.571907 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.571866    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5" event={"ID":"859207ed-804c-4ab1-bf88-2b745e381487","Type":"ContainerStarted","Data":"d2b3c28b28ebff57c7c68fb1ec63f8b4d3b86292859823cefa3550d95e69c9ed"}
Jul 03 20:45:41.572691 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.572667    2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9sbzt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 03 20:45:41.572814 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.572720    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 03 20:45:41.691510 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.691470    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fc5fe7c-f029-48eb-89f7-280833e21ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.693865 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.693842    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fc5fe7c-f029-48eb-89f7-280833e21ba2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bxf2b\" (UID: \"8fc5fe7c-f029-48eb-89f7-280833e21ba2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.768021 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.767989    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"]
Jul 03 20:45:41.772749 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.772729    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.777965 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.777946    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mvfsg\""
Jul 03 20:45:41.800181 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.800154    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"]
Jul 03 20:45:41.838117 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.838043    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:41.875648 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.875619    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:41.892970 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.892938    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893069 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.892981    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893069 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893051    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893146 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893117    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893193 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893175    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgs4x\" (UniqueName: \"kubernetes.io/projected/b5cfadb5-2d8b-410a-905f-518862b2d5b3-kube-api-access-jgs4x\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893247 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893280 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893255    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893330 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893292    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.893399 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.893382    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.995875 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.995650    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.995875 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.995712    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.996369 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.995854    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.996369 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.996146    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.996868 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.996823    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.997317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.997278    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.997547 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.996188    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.997616 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.997583    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.997672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.997613    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.997672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.997647    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:41.998268 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.997653    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.000068 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.999807    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.000068 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.999862    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgs4x\" (UniqueName: \"kubernetes.io/projected/b5cfadb5-2d8b-410a-905f-518862b2d5b3-kube-api-access-jgs4x\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.000068 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:41.999966    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.001721 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.001248    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.002221 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.002159    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.012748 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.012719    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:42.014817 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.014768    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgs4x\" (UniqueName: \"kubernetes.io/projected/b5cfadb5-2d8b-410a-905f-518862b2d5b3-kube-api-access-jgs4x\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.015111 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.014854    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cfadb5-2d8b-410a-905f-518862b2d5b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678l44k5\" (UID: \"b5cfadb5-2d8b-410a-905f-518862b2d5b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.030315 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.030290    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"]
Jul 03 20:45:42.033772 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:45:42.033745    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8fc5fe7c_f029_48eb_89f7_280833e21ba2.slice/crio-ca17e8a13747621adb5bd0452c1e81f34c1bda7f8416cac221729aebdcd5b63d WatchSource:0}: Error finding container ca17e8a13747621adb5bd0452c1e81f34c1bda7f8416cac221729aebdcd5b63d: Status 404 returned error can't find the container with id ca17e8a13747621adb5bd0452c1e81f34c1bda7f8416cac221729aebdcd5b63d
Jul 03 20:45:42.083819 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.083783    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:42.122311 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.122249    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:42.228388 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.228354    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"]
Jul 03 20:45:42.578207 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.578170    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5" event={"ID":"b5cfadb5-2d8b-410a-905f-518862b2d5b3","Type":"ContainerStarted","Data":"c4d13404a3e10da1cc4ac184593f8706cebb2b2ce420f83b6384818f378a85e6"}
Jul 03 20:45:42.579530 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.579499    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b" event={"ID":"8fc5fe7c-f029-48eb-89f7-280833e21ba2","Type":"ContainerStarted","Data":"ca17e8a13747621adb5bd0452c1e81f34c1bda7f8416cac221729aebdcd5b63d"}
Jul 03 20:45:42.754357 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.754326    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:42.856219 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.856142    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:42.970807 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:42.970774    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:43.156839 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:43.156148    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"]
Jul 03 20:45:43.156839 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:43.156388    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerName="discovery" containerID="cri-o://b5a50cce6416da2acc06f9fa9502dcdd9564970c68a4829f7d457c828ed6c9f8" gracePeriod=30
Jul 03 20:45:43.169295 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:43.169228    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:43.589228 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:43.589179    2553 generic.go:356] "Generic (PLEG): container finished" podID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerID="b5a50cce6416da2acc06f9fa9502dcdd9564970c68a4829f7d457c828ed6c9f8" exitCode=0
Jul 03 20:45:43.589398 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:43.589250    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" event={"ID":"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446","Type":"ContainerDied","Data":"b5a50cce6416da2acc06f9fa9502dcdd9564970c68a4829f7d457c828ed6c9f8"}
Jul 03 20:45:44.062914 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.062866    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:44.221494 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221463    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-ca-configmap\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221494 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221500    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-cacerts\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221530    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8pl8\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-kube-api-access-j8pl8\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221572    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-local-certs\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221587    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-token\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221603    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-kubeconfig\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221648    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-dns-cert\") pod \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\" (UID: \"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446\") "
Jul 03 20:45:44.221979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.221928    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 20:45:44.224616 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.224583    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-local-certs" (OuterVolumeSpecName: "local-certs") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 20:45:44.224741 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.224698    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-cacerts" (OuterVolumeSpecName: "cacerts") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 20:45:44.224859 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.224817    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:45:44.224936 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.224878    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 20:45:44.225184 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.225139    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 20:45:44.225976 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.225950    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-kube-api-access-j8pl8" (OuterVolumeSpecName: "kube-api-access-j8pl8") pod "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" (UID: "9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446"). InnerVolumeSpecName "kube-api-access-j8pl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:45:44.322742 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322654    2553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-ca-configmap\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.322742 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322691    2553 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-cacerts\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.322742 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322707    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pl8\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-kube-api-access-j8pl8\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.322742 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322721    2553 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-local-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.322742 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322734    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-token\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.323099 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322747    2553 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-kubeconfig\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.323099 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.322760    2553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446-istio-csr-dns-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:45:44.596248 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.596167    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"
Jul 03 20:45:44.596248 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.596175    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt" event={"ID":"9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446","Type":"ContainerDied","Data":"aa574df0797a679c5b6ada0f37e64cfb6dede651e15eeb83b51d6dee92a8da05"}
Jul 03 20:45:44.596474 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.596266    2553 scope.go:117] "RemoveContainer" containerID="b5a50cce6416da2acc06f9fa9502dcdd9564970c68a4829f7d457c828ed6c9f8"
Jul 03 20:45:44.599768 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.599739    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5" event={"ID":"859207ed-804c-4ab1-bf88-2b745e381487","Type":"ContainerStarted","Data":"c18f500ce98a63565078698d1cb2ec432afff82338a3676005bfd2c1593b399e"}
Jul 03 20:45:44.599952 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.599929    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:45:44.643783 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.643708    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5" podStartSLOduration=1.9493179980000002 podStartE2EDuration="4.643687304s" totalImagesPullingTime="2.694369306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:44.641215484 +0000 UTC m=+516.721956874" watchObservedRunningTime="2026-07-03 20:45:44.643687304 +0000 UTC m=+516.724428715"
Jul 03 20:45:44.667218 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.667176    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"]
Jul 03 20:45:44.681352 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.681322    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9sbzt"]
Jul 03 20:45:44.871649 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:44.871581    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" path="/var/lib/kubelet/pods/9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446/volumes"
Jul 03 20:45:45.606335 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:45.606290    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b" event={"ID":"8fc5fe7c-f029-48eb-89f7-280833e21ba2","Type":"ContainerStarted","Data":"fd21e87f9c3d17f3751c00749a9606559d48168fdab275532e59d3eeefcf400c"}
Jul 03 20:45:45.606794 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:45.606372    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:45:45.624917 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:45.624830    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b" podStartSLOduration=2.686914766 podStartE2EDuration="5.624763067s" totalImagesPullingTime="2.937848301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:45.623255402 +0000 UTC m=+517.703996809" watchObservedRunningTime="2026-07-03 20:45:45.624763067 +0000 UTC m=+517.705504477"
Jul 03 20:45:46.937233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:46.937178    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:45:46.937542 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:46.937287    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:45:46.937542 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:46.937330    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:45:47.555381 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:47.555340    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8fsbz"
Jul 03 20:45:47.557433 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:47.557413    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-92l6h"
Jul 03 20:45:47.619018 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:47.618979    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5" event={"ID":"b5cfadb5-2d8b-410a-905f-518862b2d5b3","Type":"ContainerStarted","Data":"d4783998222b18c620f5c037fe2ad93ff2a271d0c5477c81041566a1a275ce85"}
Jul 03 20:45:47.642832 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:47.642767    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5" podStartSLOduration=1.948055511 podStartE2EDuration="6.642746549s" totalImagesPullingTime="4.694691038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:45:47.64130711 +0000 UTC m=+519.722048517" watchObservedRunningTime="2026-07-03 20:45:47.642746549 +0000 UTC m=+519.723487958"
Jul 03 20:45:48.084230 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:48.084198    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:48.088974 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:48.088946    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:48.624040 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:48.624010    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:48.625087 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:48.625063    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678l44k5"
Jul 03 20:45:50.109587 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:50.109527    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:45:50.210069 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:45:50.210041    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:46:15.612835 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:15.612805    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-v6cj5"
Jul 03 20:46:16.613805 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:16.613770    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bxf2b"
Jul 03 20:46:31.124946 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.124885    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:46:31.225192 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.225160    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:46:31.245820 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.245787    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9xcf6"]
Jul 03 20:46:31.246123 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.246108    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerName="discovery"
Jul 03 20:46:31.246169 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.246124    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerName="discovery"
Jul 03 20:46:31.246210 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.246182    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8e5d5c-ca74-4add-8cd5-3c02fa2f7446" containerName="discovery"
Jul 03 20:46:31.257458 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.257431    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:31.262173 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.262149    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 03 20:46:31.262490 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.262399    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 03 20:46:31.262623 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.262408    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xkm2x\""
Jul 03 20:46:31.265801 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.265769    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9xcf6"]
Jul 03 20:46:31.328859 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.328828    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:46:31.400161 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.400072    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll654\" (UniqueName: \"kubernetes.io/projected/dbc92272-ea7b-41b5-8bc2-15847024409e-kube-api-access-ll654\") pod \"kuadrant-operator-catalog-9xcf6\" (UID: \"dbc92272-ea7b-41b5-8bc2-15847024409e\") " pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:31.500909 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.500869    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll654\" (UniqueName: \"kubernetes.io/projected/dbc92272-ea7b-41b5-8bc2-15847024409e-kube-api-access-ll654\") pod \"kuadrant-operator-catalog-9xcf6\" (UID: \"dbc92272-ea7b-41b5-8bc2-15847024409e\") " pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:31.510999 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.510970    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll654\" (UniqueName: \"kubernetes.io/projected/dbc92272-ea7b-41b5-8bc2-15847024409e-kube-api-access-ll654\") pod \"kuadrant-operator-catalog-9xcf6\" (UID: \"dbc92272-ea7b-41b5-8bc2-15847024409e\") " pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:31.572007 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.571974    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:31.596587 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.596555    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9xcf6"]
Jul 03 20:46:31.731036 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.731008    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9xcf6"]
Jul 03 20:46:31.733205 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:46:31.733164    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc92272_ea7b_41b5_8bc2_15847024409e.slice/crio-135e3926f69c5f739c9e249b26b002ea2039a1ff19c6744773ff0865e2f43daa WatchSource:0}: Error finding container 135e3926f69c5f739c9e249b26b002ea2039a1ff19c6744773ff0865e2f43daa: Status 404 returned error can't find the container with id 135e3926f69c5f739c9e249b26b002ea2039a1ff19c6744773ff0865e2f43daa
Jul 03 20:46:31.798032 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.797997    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x5fk2"]
Jul 03 20:46:31.801483 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.801460    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:31.802129 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.802099    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6" event={"ID":"dbc92272-ea7b-41b5-8bc2-15847024409e","Type":"ContainerStarted","Data":"135e3926f69c5f739c9e249b26b002ea2039a1ff19c6744773ff0865e2f43daa"}
Jul 03 20:46:31.812867 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.812845    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x5fk2"]
Jul 03 20:46:31.904749 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:31.904661    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2vh\" (UniqueName: \"kubernetes.io/projected/31977d5a-0150-4f3c-a532-9340ff04ac16-kube-api-access-hp2vh\") pod \"kuadrant-operator-catalog-x5fk2\" (UID: \"31977d5a-0150-4f3c-a532-9340ff04ac16\") " pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:32.005823 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:32.005791    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2vh\" (UniqueName: \"kubernetes.io/projected/31977d5a-0150-4f3c-a532-9340ff04ac16-kube-api-access-hp2vh\") pod \"kuadrant-operator-catalog-x5fk2\" (UID: \"31977d5a-0150-4f3c-a532-9340ff04ac16\") " pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:32.013963 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:32.013934    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2vh\" (UniqueName: \"kubernetes.io/projected/31977d5a-0150-4f3c-a532-9340ff04ac16-kube-api-access-hp2vh\") pod \"kuadrant-operator-catalog-x5fk2\" (UID: \"31977d5a-0150-4f3c-a532-9340ff04ac16\") " pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:32.110958 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:32.110928    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:32.254138 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:32.254013    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-x5fk2"]
Jul 03 20:46:32.256593 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:46:32.256566    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31977d5a_0150_4f3c_a532_9340ff04ac16.slice/crio-47445a75e1cab559385161a348254a2603216a48c24f5789194f7ff34422f283 WatchSource:0}: Error finding container 47445a75e1cab559385161a348254a2603216a48c24f5789194f7ff34422f283: Status 404 returned error can't find the container with id 47445a75e1cab559385161a348254a2603216a48c24f5789194f7ff34422f283
Jul 03 20:46:32.807590 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:32.807556    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2" event={"ID":"31977d5a-0150-4f3c-a532-9340ff04ac16","Type":"ContainerStarted","Data":"47445a75e1cab559385161a348254a2603216a48c24f5789194f7ff34422f283"}
Jul 03 20:46:33.813992 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:33.813947    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6" event={"ID":"dbc92272-ea7b-41b5-8bc2-15847024409e","Type":"ContainerStarted","Data":"64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37"}
Jul 03 20:46:33.814642 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:33.813994    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6" podUID="dbc92272-ea7b-41b5-8bc2-15847024409e" containerName="registry-server" containerID="cri-o://64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37" gracePeriod=2
Jul 03 20:46:33.815545 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:33.815520    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2" event={"ID":"31977d5a-0150-4f3c-a532-9340ff04ac16","Type":"ContainerStarted","Data":"1d118487593f9b62847e210630a3ca9897a8c63d4801e9fefaa55fe632de45f1"}
Jul 03 20:46:33.833752 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:33.833607    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6" podStartSLOduration=0.944214743 podStartE2EDuration="2.833592753s" totalImagesPullingTime="1.88937801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:46:33.833209315 +0000 UTC m=+565.913950721" watchObservedRunningTime="2026-07-03 20:46:33.833592753 +0000 UTC m=+565.914334160"
Jul 03 20:46:33.849211 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:33.849160    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2" podStartSLOduration=1.482143251 podStartE2EDuration="2.849145509s" totalImagesPullingTime="1.367002258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:46:33.84847079 +0000 UTC m=+565.929212198" watchObservedRunningTime="2026-07-03 20:46:33.849145509 +0000 UTC m=+565.929886934"
Jul 03 20:46:34.055417 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.055387    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:34.126637 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.126601    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll654\" (UniqueName: \"kubernetes.io/projected/dbc92272-ea7b-41b5-8bc2-15847024409e-kube-api-access-ll654\") pod \"dbc92272-ea7b-41b5-8bc2-15847024409e\" (UID: \"dbc92272-ea7b-41b5-8bc2-15847024409e\") "
Jul 03 20:46:34.128697 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.128677    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc92272-ea7b-41b5-8bc2-15847024409e-kube-api-access-ll654" (OuterVolumeSpecName: "kube-api-access-ll654") pod "dbc92272-ea7b-41b5-8bc2-15847024409e" (UID: "dbc92272-ea7b-41b5-8bc2-15847024409e"). InnerVolumeSpecName "kube-api-access-ll654". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:46:34.228102 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.228031    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll654\" (UniqueName: \"kubernetes.io/projected/dbc92272-ea7b-41b5-8bc2-15847024409e-kube-api-access-ll654\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:46:34.820886 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.820845    2553 generic.go:356] "Generic (PLEG): container finished" podID="dbc92272-ea7b-41b5-8bc2-15847024409e" containerID="64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37" exitCode=0
Jul 03 20:46:34.821281 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.820912    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6"
Jul 03 20:46:34.821281 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.820936    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6" event={"ID":"dbc92272-ea7b-41b5-8bc2-15847024409e","Type":"ContainerDied","Data":"64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37"}
Jul 03 20:46:34.821281 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.820993    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9xcf6" event={"ID":"dbc92272-ea7b-41b5-8bc2-15847024409e","Type":"ContainerDied","Data":"135e3926f69c5f739c9e249b26b002ea2039a1ff19c6744773ff0865e2f43daa"}
Jul 03 20:46:34.821281 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.821015    2553 scope.go:117] "RemoveContainer" containerID="64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37"
Jul 03 20:46:34.830274 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.830256    2553 scope.go:117] "RemoveContainer" containerID="64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37"
Jul 03 20:46:34.830530 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:46:34.830497    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37\": container with ID starting with 64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37 not found: ID does not exist" containerID="64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37"
Jul 03 20:46:34.830591 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.830535    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37"} err="failed to get container status \"64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37\": rpc error: code = NotFound desc = could not find container \"64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37\": container with ID starting with 64d83e7007d6ac052a62d592939d8f3a4fb6fd1ceae46f4af32eb59f062c5d37 not found: ID does not exist"
Jul 03 20:46:34.842379 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.842358    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9xcf6"]
Jul 03 20:46:34.843874 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.843855    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9xcf6"]
Jul 03 20:46:34.869644 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:34.869619    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc92272-ea7b-41b5-8bc2-15847024409e" path="/var/lib/kubelet/pods/dbc92272-ea7b-41b5-8bc2-15847024409e/volumes"
Jul 03 20:46:42.111383 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:42.111347    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:42.111766 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:42.111428    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:42.133809 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:42.133785    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:42.878282 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:42.878258    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-x5fk2"
Jul 03 20:46:59.329619 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:46:59.329588    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:00.567851 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:00.567818    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:03.456864 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.456779    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"]
Jul 03 20:47:03.457392 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.457220    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc92272-ea7b-41b5-8bc2-15847024409e" containerName="registry-server"
Jul 03 20:47:03.457392 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.457237    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc92272-ea7b-41b5-8bc2-15847024409e" containerName="registry-server"
Jul 03 20:47:03.460095 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.457880    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc92272-ea7b-41b5-8bc2-15847024409e" containerName="registry-server"
Jul 03 20:47:03.462439 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.462409    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:03.466728 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.466700    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6b6qf\""
Jul 03 20:47:03.466848 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.466826    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgdf\" (UniqueName: \"kubernetes.io/projected/41637a53-f3b5-4e15-9983-937e9ed32138-kube-api-access-gzgdf\") pod \"authorino-operator-75cc8d5cb7-rs5vq\" (UID: \"41637a53-f3b5-4e15-9983-937e9ed32138\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:03.479642 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.479619    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"]
Jul 03 20:47:03.568291 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.568255    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgdf\" (UniqueName: \"kubernetes.io/projected/41637a53-f3b5-4e15-9983-937e9ed32138-kube-api-access-gzgdf\") pod \"authorino-operator-75cc8d5cb7-rs5vq\" (UID: \"41637a53-f3b5-4e15-9983-937e9ed32138\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:03.583297 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.583264    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgdf\" (UniqueName: \"kubernetes.io/projected/41637a53-f3b5-4e15-9983-937e9ed32138-kube-api-access-gzgdf\") pod \"authorino-operator-75cc8d5cb7-rs5vq\" (UID: \"41637a53-f3b5-4e15-9983-937e9ed32138\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:03.772748 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.772720    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:03.926129 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.925953    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"]
Jul 03 20:47:03.928837 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:03.928803    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41637a53_f3b5_4e15_9983_937e9ed32138.slice/crio-a9cf9b737ad3e8f05886f692929d7a529428a3a61bfd75559040e30c987b3af7 WatchSource:0}: Error finding container a9cf9b737ad3e8f05886f692929d7a529428a3a61bfd75559040e30c987b3af7: Status 404 returned error can't find the container with id a9cf9b737ad3e8f05886f692929d7a529428a3a61bfd75559040e30c987b3af7
Jul 03 20:47:03.937497 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:03.937474    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq" event={"ID":"41637a53-f3b5-4e15-9983-937e9ed32138","Type":"ContainerStarted","Data":"a9cf9b737ad3e8f05886f692929d7a529428a3a61bfd75559040e30c987b3af7"}
Jul 03 20:47:04.844266 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:04.844177    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:06.700450 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.700415    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"]
Jul 03 20:47:06.703620 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.703603    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:06.707328 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.707305    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qr5jm\""
Jul 03 20:47:06.707578 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.707563    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 03 20:47:06.725053 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.725028    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"]
Jul 03 20:47:06.790442 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.790414    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44g2z\" (UniqueName: \"kubernetes.io/projected/0c44cc94-1784-4508-b007-3a12a7887e5a-kube-api-access-44g2z\") pod \"dns-operator-controller-manager-864dd78c4c-kzv4h\" (UID: \"0c44cc94-1784-4508-b007-3a12a7887e5a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:06.869370 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.869335    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:06.891421 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.891386    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44g2z\" (UniqueName: \"kubernetes.io/projected/0c44cc94-1784-4508-b007-3a12a7887e5a-kube-api-access-44g2z\") pod \"dns-operator-controller-manager-864dd78c4c-kzv4h\" (UID: \"0c44cc94-1784-4508-b007-3a12a7887e5a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:06.904045 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.904007    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44g2z\" (UniqueName: \"kubernetes.io/projected/0c44cc94-1784-4508-b007-3a12a7887e5a-kube-api-access-44g2z\") pod \"dns-operator-controller-manager-864dd78c4c-kzv4h\" (UID: \"0c44cc94-1784-4508-b007-3a12a7887e5a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:06.953758 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.953675    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq" event={"ID":"41637a53-f3b5-4e15-9983-937e9ed32138","Type":"ContainerStarted","Data":"cd6b09ab2a6acf6ba160c223e4797d539477a0a0c96f9f6395cbcbb51f0ccab0"}
Jul 03 20:47:06.953889 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.953763    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:06.986676 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:06.986628    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq" podStartSLOduration=1.873260435 podStartE2EDuration="3.986614706s" totalImagesPullingTime="2.113354271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:06.985888104 +0000 UTC m=+599.066629513" watchObservedRunningTime="2026-07-03 20:47:06.986614706 +0000 UTC m=+599.067356114"
Jul 03 20:47:07.013585 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:07.013561    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:07.038724 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:07.038634    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:07.202558 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:07.202535    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"]
Jul 03 20:47:07.203869 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:07.203808    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c44cc94_1784_4508_b007_3a12a7887e5a.slice/crio-1fe421989a3f1ced056b5ea153ec76fe874945a62b075b16a98cc86a7de85d4d WatchSource:0}: Error finding container 1fe421989a3f1ced056b5ea153ec76fe874945a62b075b16a98cc86a7de85d4d: Status 404 returned error can't find the container with id 1fe421989a3f1ced056b5ea153ec76fe874945a62b075b16a98cc86a7de85d4d
Jul 03 20:47:07.794647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:07.794609    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:07.961908 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:07.961857    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h" event={"ID":"0c44cc94-1784-4508-b007-3a12a7887e5a","Type":"ContainerStarted","Data":"1fe421989a3f1ced056b5ea153ec76fe874945a62b075b16a98cc86a7de85d4d"}
Jul 03 20:47:08.636447 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:08.636334    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:08.936054 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:08.936033    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 03 20:47:09.971577 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:09.971539    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h" event={"ID":"0c44cc94-1784-4508-b007-3a12a7887e5a","Type":"ContainerStarted","Data":"9e09870045279532ab1f83850f23df1e2d903cfd1da4872dacdeac4284134908"}
Jul 03 20:47:09.973833 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:09.971728    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:09.989636 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:09.989590    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h" podStartSLOduration=2.262988133 podStartE2EDuration="3.989572011s" totalImagesPullingTime="1.726583878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:09.988469007 +0000 UTC m=+602.069210421" watchObservedRunningTime="2026-07-03 20:47:09.989572011 +0000 UTC m=+602.070313419"
Jul 03 20:47:10.120064 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.120029    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"]
Jul 03 20:47:10.123198 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.123168    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:10.129361 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.129298    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qbj9f\""
Jul 03 20:47:10.138211 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.138184    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"]
Jul 03 20:47:10.218721 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.218690    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnxw\" (UniqueName: \"kubernetes.io/projected/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a-kube-api-access-tpnxw\") pod \"limitador-operator-controller-manager-859d7b78bd-g7zck\" (UID: \"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:10.319076 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.319044    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnxw\" (UniqueName: \"kubernetes.io/projected/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a-kube-api-access-tpnxw\") pod \"limitador-operator-controller-manager-859d7b78bd-g7zck\" (UID: \"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:10.333854 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.333835    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnxw\" (UniqueName: \"kubernetes.io/projected/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a-kube-api-access-tpnxw\") pod \"limitador-operator-controller-manager-859d7b78bd-g7zck\" (UID: \"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:10.434975 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.434941    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:10.569702 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.569603    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"]
Jul 03 20:47:10.575630 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:10.575606    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3af8dd_6bf9_4327_9ffe_8cd242e6267a.slice/crio-4fc06a0dc2fb4ed426719beb4992f44c23b09bc289425c91025805e477b3623c WatchSource:0}: Error finding container 4fc06a0dc2fb4ed426719beb4992f44c23b09bc289425c91025805e477b3623c: Status 404 returned error can't find the container with id 4fc06a0dc2fb4ed426719beb4992f44c23b09bc289425c91025805e477b3623c
Jul 03 20:47:10.726907 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.726870    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:10.979177 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:10.979080    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck" event={"ID":"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a","Type":"ContainerStarted","Data":"4fc06a0dc2fb4ed426719beb4992f44c23b09bc289425c91025805e477b3623c"}
Jul 03 20:47:11.206063 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:11.206026    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:11.985088 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:11.985060    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck" event={"ID":"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a","Type":"ContainerStarted","Data":"297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2"}
Jul 03 20:47:11.985385 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:11.985125    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:12.006467 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:12.006424    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck" podStartSLOduration=0.670218097 podStartE2EDuration="2.006409621s" totalImagesPullingTime="1.336191524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:12.004400402 +0000 UTC m=+604.085141821" watchObservedRunningTime="2026-07-03 20:47:12.006409621 +0000 UTC m=+604.087151087"
Jul 03 20:47:14.420524 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.420495    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:14.930115 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.930077    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"]
Jul 03 20:47:14.933461 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.933439    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:14.936365 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.936343    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tc8gn\""
Jul 03 20:47:14.948070 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.948046    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"]
Jul 03 20:47:14.956333 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.956311    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:14.956444 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:14.956357    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxj5\" (UniqueName: \"kubernetes.io/projected/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-kube-api-access-4kxj5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:15.057080 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.057047    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:15.057256 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.057100    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxj5\" (UniqueName: \"kubernetes.io/projected/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-kube-api-access-4kxj5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:15.057457 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.057439    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:15.065649 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.065621    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxj5\" (UniqueName: \"kubernetes.io/projected/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-kube-api-access-4kxj5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:15.243241 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.243159    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:15.377136 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.377108    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"]
Jul 03 20:47:15.379046 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:15.379005    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode830d277_0b6d_4bfa_8953_4c9a3f0a5957.slice/crio-e3505bf2aea8b40ee59077ef4f9aab54ea81f254e0ed42d293687c57ed8340f5 WatchSource:0}: Error finding container e3505bf2aea8b40ee59077ef4f9aab54ea81f254e0ed42d293687c57ed8340f5: Status 404 returned error can't find the container with id e3505bf2aea8b40ee59077ef4f9aab54ea81f254e0ed42d293687c57ed8340f5
Jul 03 20:47:15.381504 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:15.381490    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 03 20:47:16.003677 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:16.003644    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl" event={"ID":"e830d277-0b6d-4bfa-8953-4c9a3f0a5957","Type":"ContainerStarted","Data":"e3505bf2aea8b40ee59077ef4f9aab54ea81f254e0ed42d293687c57ed8340f5"}
Jul 03 20:47:16.318751 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:16.318716    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:16.418863 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:16.418828    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:16.728261 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:16.728171    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:16.828360 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:16.828316    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:17.964661 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:17.964635    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-rs5vq"
Jul 03 20:47:20.151999 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:20.151962    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:20.252310 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:20.252264    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:20.613574 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:20.613542    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:20.678800 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:20.678768    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:20.982102 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:20.982023    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzv4h"
Jul 03 20:47:22.034519 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:22.034486    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl" event={"ID":"e830d277-0b6d-4bfa-8953-4c9a3f0a5957","Type":"ContainerStarted","Data":"50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6"}
Jul 03 20:47:22.034915 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:22.034595    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:22.055714 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:22.055660    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl" podStartSLOduration=2.140118629 podStartE2EDuration="8.055645044s" totalImagesPullingTime="5.915526415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:22.05319182 +0000 UTC m=+614.133933227" watchObservedRunningTime="2026-07-03 20:47:22.055645044 +0000 UTC m=+614.136386451"
Jul 03 20:47:22.992215 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:22.992186    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:24.097152 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:24.097119    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:24.376276 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:24.376199    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:24.471688 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:24.471660    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:27.458744 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:27.458710    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:27.559780 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:27.559742    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:31.893688 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:31.893611    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:31.943486 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:31.943454    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:31.999233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:31.999197    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:32.100644 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:32.100617    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:33.042148 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:33.042117    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:34.653062 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.653032    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"]
Jul 03 20:47:34.653582 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.653203    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" containerName="manager" containerID="cri-o://50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6" gracePeriod=2
Jul 03 20:47:34.661866 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.661842    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"]
Jul 03 20:47:34.665113 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.665080    2553 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\": pods \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:34.685963 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.685904    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"]
Jul 03 20:47:34.686216 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.686139    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" containerName="manager" containerID="cri-o://297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2" gracePeriod=2
Jul 03 20:47:34.687567 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.687534    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"]
Jul 03 20:47:34.688167 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.688144    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" containerName="manager"
Jul 03 20:47:34.688167 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.688168    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" containerName="manager"
Jul 03 20:47:34.688292 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.688241    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" containerName="manager"
Jul 03 20:47:34.688778 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.688743    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" is forbidden: User \"system:node:ip-10-0-129-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-236.ec2.internal' and this object" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:34.691478 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.691460    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"]
Jul 03 20:47:34.691576 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.691537    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.692698 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.692656    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g7zck\" is forbidden: User \"system:node:ip-10-0-129-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-236.ec2.internal' and this object" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:34.703084 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.703062    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"]
Jul 03 20:47:34.706680 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.706662    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"]
Jul 03 20:47:34.707015 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.707000    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" containerName="manager"
Jul 03 20:47:34.707098 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.707018    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" containerName="manager"
Jul 03 20:47:34.707151 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.707130    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" containerName="manager"
Jul 03 20:47:34.709993 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.709963    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:34.720372 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.720194    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" is forbidden: User \"system:node:ip-10-0-129-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-236.ec2.internal' and this object" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:34.722391 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.722362    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"]
Jul 03 20:47:34.722493 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.722388    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g7zck\" is forbidden: User \"system:node:ip-10-0-129-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-236.ec2.internal' and this object" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:34.724522 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.724493    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-g7zck\" is forbidden: User \"system:node:ip-10-0-129-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-236.ec2.internal' and this object" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:34.744223 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:34.744191    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5vxtl\" is forbidden: User \"system:node:ip-10-0-129-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-236.ec2.internal' and this object" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:34.830204 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.830167    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fgmjh\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.830204 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.830210    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz5sd\" (UniqueName: \"kubernetes.io/projected/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-kube-api-access-rz5sd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fgmjh\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.830432 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.830271    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlcx\" (UniqueName: \"kubernetes.io/projected/c861b0e8-54fe-4ad3-828f-66ef4a50a9dd-kube-api-access-wjlcx\") pod \"limitador-operator-controller-manager-859d7b78bd-jp5rq\" (UID: \"c861b0e8-54fe-4ad3-828f-66ef4a50a9dd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:34.923424 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.923400    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:34.926128 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.926110    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:34.931653 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.931632    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fgmjh\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.931716 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.931670    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz5sd\" (UniqueName: \"kubernetes.io/projected/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-kube-api-access-rz5sd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fgmjh\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.931716 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.931699    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlcx\" (UniqueName: \"kubernetes.io/projected/c861b0e8-54fe-4ad3-828f-66ef4a50a9dd-kube-api-access-wjlcx\") pod \"limitador-operator-controller-manager-859d7b78bd-jp5rq\" (UID: \"c861b0e8-54fe-4ad3-828f-66ef4a50a9dd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:34.932016 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.932001    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fgmjh\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.940137 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.940117    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz5sd\" (UniqueName: \"kubernetes.io/projected/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-kube-api-access-rz5sd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fgmjh\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:34.940329 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:34.940310    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlcx\" (UniqueName: \"kubernetes.io/projected/c861b0e8-54fe-4ad3-828f-66ef4a50a9dd-kube-api-access-wjlcx\") pod \"limitador-operator-controller-manager-859d7b78bd-jp5rq\" (UID: \"c861b0e8-54fe-4ad3-828f-66ef4a50a9dd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:35.032929 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.032881    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpnxw\" (UniqueName: \"kubernetes.io/projected/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a-kube-api-access-tpnxw\") pod \"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a\" (UID: \"2f3af8dd-6bf9-4327-9ffe-8cd242e6267a\") "
Jul 03 20:47:35.033078 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.032940    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-extensions-socket-volume\") pod \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") "
Jul 03 20:47:35.033078 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.033056    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kxj5\" (UniqueName: \"kubernetes.io/projected/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-kube-api-access-4kxj5\") pod \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\" (UID: \"e830d277-0b6d-4bfa-8953-4c9a3f0a5957\") "
Jul 03 20:47:35.033482 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.033457    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e830d277-0b6d-4bfa-8953-4c9a3f0a5957" (UID: "e830d277-0b6d-4bfa-8953-4c9a3f0a5957"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 20:47:35.035000 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.034976    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-kube-api-access-4kxj5" (OuterVolumeSpecName: "kube-api-access-4kxj5") pod "e830d277-0b6d-4bfa-8953-4c9a3f0a5957" (UID: "e830d277-0b6d-4bfa-8953-4c9a3f0a5957"). InnerVolumeSpecName "kube-api-access-4kxj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:47:35.035078 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.035006    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a-kube-api-access-tpnxw" (OuterVolumeSpecName: "kube-api-access-tpnxw") pod "2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" (UID: "2f3af8dd-6bf9-4327-9ffe-8cd242e6267a"). InnerVolumeSpecName "kube-api-access-tpnxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:47:35.091374 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.091343    2553 generic.go:356] "Generic (PLEG): container finished" podID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" containerID="297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2" exitCode=0
Jul 03 20:47:35.091502 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.091395    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g7zck"
Jul 03 20:47:35.091502 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.091445    2553 scope.go:117] "RemoveContainer" containerID="297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2"
Jul 03 20:47:35.092924 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.092881    2553 generic.go:356] "Generic (PLEG): container finished" podID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" containerID="50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6" exitCode=0
Jul 03 20:47:35.093021 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.092942    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5vxtl"
Jul 03 20:47:35.093206 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.093193    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:35.098933 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.098888    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:35.100156 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.100094    2553 scope.go:117] "RemoveContainer" containerID="297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2"
Jul 03 20:47:35.100377 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:35.100355    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2\": container with ID starting with 297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2 not found: ID does not exist" containerID="297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2"
Jul 03 20:47:35.100506 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.100439    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2"} err="failed to get container status \"297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2\": rpc error: code = NotFound desc = could not find container \"297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2\": container with ID starting with 297c0ca56ed930e2bcdfa404e93ae3b75e5bc80865c2d6d6a7010529d1a869d2 not found: ID does not exist"
Jul 03 20:47:35.100506 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.100466    2553 scope.go:117] "RemoveContainer" containerID="50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6"
Jul 03 20:47:35.116871 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.116851    2553 scope.go:117] "RemoveContainer" containerID="50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6"
Jul 03 20:47:35.117170 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:35.117149    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6\": container with ID starting with 50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6 not found: ID does not exist" containerID="50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6"
Jul 03 20:47:35.117264 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.117175    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6"} err="failed to get container status \"50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6\": rpc error: code = NotFound desc = could not find container \"50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6\": container with ID starting with 50f26ceec85e70cc87167022ffacc112034e3c7c79ae801b825b0c584cdd2af6 not found: ID does not exist"
Jul 03 20:47:35.134663 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.134633    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxj5\" (UniqueName: \"kubernetes.io/projected/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-kube-api-access-4kxj5\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:47:35.134663 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.134660    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpnxw\" (UniqueName: \"kubernetes.io/projected/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a-kube-api-access-tpnxw\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:47:35.134663 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.134669    2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e830d277-0b6d-4bfa-8953-4c9a3f0a5957-extensions-socket-volume\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:47:35.240563 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.240539    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"]
Jul 03 20:47:35.243563 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:35.243534    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6af1bd_25bf_4934_a4da_c9e3d2f943ec.slice/crio-a5dfe6faa1f74860ba1e8ebd329f0644cc7802b8e2906d4f6e1269e9b95277b7 WatchSource:0}: Error finding container a5dfe6faa1f74860ba1e8ebd329f0644cc7802b8e2906d4f6e1269e9b95277b7: Status 404 returned error can't find the container with id a5dfe6faa1f74860ba1e8ebd329f0644cc7802b8e2906d4f6e1269e9b95277b7
Jul 03 20:47:35.259169 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.259072    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"]
Jul 03 20:47:35.261561 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:35.261541    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc861b0e8_54fe_4ad3_828f_66ef4a50a9dd.slice/crio-98a135e8decf98da9773861137973de0c9e1d668dca61dee56552a560615fc15 WatchSource:0}: Error finding container 98a135e8decf98da9773861137973de0c9e1d668dca61dee56552a560615fc15: Status 404 returned error can't find the container with id 98a135e8decf98da9773861137973de0c9e1d668dca61dee56552a560615fc15
Jul 03 20:47:35.529426 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.529401    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:35.620235 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:35.620201    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:36.100402 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.100362    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh" event={"ID":"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec","Type":"ContainerStarted","Data":"ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220"}
Jul 03 20:47:36.100868 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.100453    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh" event={"ID":"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec","Type":"ContainerStarted","Data":"a5dfe6faa1f74860ba1e8ebd329f0644cc7802b8e2906d4f6e1269e9b95277b7"}
Jul 03 20:47:36.100868 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.100484    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:36.103696 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.103668    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq" event={"ID":"c861b0e8-54fe-4ad3-828f-66ef4a50a9dd","Type":"ContainerStarted","Data":"f64ba5c9478cbf97a13d4440f3da8ba05ccad2fb0772def087992ed72007def8"}
Jul 03 20:47:36.103830 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.103720    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq" event={"ID":"c861b0e8-54fe-4ad3-828f-66ef4a50a9dd","Type":"ContainerStarted","Data":"98a135e8decf98da9773861137973de0c9e1d668dca61dee56552a560615fc15"}
Jul 03 20:47:36.103830 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.103753    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:36.124367 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.124314    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh" podStartSLOduration=2.124300448 podStartE2EDuration="2.124300448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:36.122226618 +0000 UTC m=+628.202968023" watchObservedRunningTime="2026-07-03 20:47:36.124300448 +0000 UTC m=+628.205041923"
Jul 03 20:47:36.124530 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.124509    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:36.146652 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.146606    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq" podStartSLOduration=2.146590874 podStartE2EDuration="2.146590874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:36.145373658 +0000 UTC m=+628.226115065" watchObservedRunningTime="2026-07-03 20:47:36.146590874 +0000 UTC m=+628.227332281"
Jul 03 20:47:36.222873 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.222838    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:36.870359 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.870316    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f3af8dd-6bf9-4327-9ffe-8cd242e6267a" path="/var/lib/kubelet/pods/2f3af8dd-6bf9-4327-9ffe-8cd242e6267a/volumes"
Jul 03 20:47:36.870749 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:36.870729    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e830d277-0b6d-4bfa-8953-4c9a3f0a5957" path="/var/lib/kubelet/pods/e830d277-0b6d-4bfa-8953-4c9a3f0a5957/volumes"
Jul 03 20:47:39.107082 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:39.107042    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:39.162846 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:39.162808    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:43.716293 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:43.716261    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:43.816120 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:43.816087    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:46.309867 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:46.309838    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:46.407069 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:46.407039    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:47.112041 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:47.112008    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jp5rq"
Jul 03 20:47:47.112211 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:47.112083    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:51.284692 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:51.284653    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:51.351264 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:51.351233    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:47:51.768961 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:51.768851    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"]
Jul 03 20:47:51.769129 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:51.769103    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh" podUID="fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" containerName="manager" containerID="cri-o://ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220" gracePeriod=10
Jul 03 20:47:52.053143 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.053108    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"]
Jul 03 20:47:52.116150 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.116115    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"]
Jul 03 20:47:52.116300 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.116214    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.280614 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.280576    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszlf\" (UniqueName: \"kubernetes.io/projected/a5daf2a5-3760-4a24-8f48-29898e1368cb-kube-api-access-gszlf\") pod \"kuadrant-operator-controller-manager-86878c769f-jtwtw\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.280614 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.280616    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5daf2a5-3760-4a24-8f48-29898e1368cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jtwtw\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.381959 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.381927    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszlf\" (UniqueName: \"kubernetes.io/projected/a5daf2a5-3760-4a24-8f48-29898e1368cb-kube-api-access-gszlf\") pod \"kuadrant-operator-controller-manager-86878c769f-jtwtw\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.382350 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.381976    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5daf2a5-3760-4a24-8f48-29898e1368cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jtwtw\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.382453 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.382431    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5daf2a5-3760-4a24-8f48-29898e1368cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jtwtw\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.390004 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.389985    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszlf\" (UniqueName: \"kubernetes.io/projected/a5daf2a5-3760-4a24-8f48-29898e1368cb-kube-api-access-gszlf\") pod \"kuadrant-operator-controller-manager-86878c769f-jtwtw\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.426500 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.426476    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:52.580451 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.580318    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"]
Jul 03 20:47:52.583844 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.583814    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:52.584528 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:47:52.584501    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5daf2a5_3760_4a24_8f48_29898e1368cb.slice/crio-96cfd3d3d14e17024fc3892c1a855c7c549f9a41cd71309bb6c3c52a181cc0a4 WatchSource:0}: Error finding container 96cfd3d3d14e17024fc3892c1a855c7c549f9a41cd71309bb6c3c52a181cc0a4: Status 404 returned error can't find the container with id 96cfd3d3d14e17024fc3892c1a855c7c549f9a41cd71309bb6c3c52a181cc0a4
Jul 03 20:47:52.684459 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.684426    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz5sd\" (UniqueName: \"kubernetes.io/projected/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-kube-api-access-rz5sd\") pod \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") "
Jul 03 20:47:52.684646 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.684514    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-extensions-socket-volume\") pod \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\" (UID: \"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec\") "
Jul 03 20:47:52.684948 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.684916    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" (UID: "fe6af1bd-25bf-4934-a4da-c9e3d2f943ec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 20:47:52.686475 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.686455    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-kube-api-access-rz5sd" (OuterVolumeSpecName: "kube-api-access-rz5sd") pod "fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" (UID: "fe6af1bd-25bf-4934-a4da-c9e3d2f943ec"). InnerVolumeSpecName "kube-api-access-rz5sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:47:52.785920 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.785876    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz5sd\" (UniqueName: \"kubernetes.io/projected/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-kube-api-access-rz5sd\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:47:52.785920 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:52.785921    2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec-extensions-socket-volume\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:47:53.188470 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.188382    2553 generic.go:356] "Generic (PLEG): container finished" podID="fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" containerID="ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220" exitCode=0
Jul 03 20:47:53.188470 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.188428    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"
Jul 03 20:47:53.188656 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.188470    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh" event={"ID":"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec","Type":"ContainerDied","Data":"ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220"}
Jul 03 20:47:53.188656 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.188523    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh" event={"ID":"fe6af1bd-25bf-4934-a4da-c9e3d2f943ec","Type":"ContainerDied","Data":"a5dfe6faa1f74860ba1e8ebd329f0644cc7802b8e2906d4f6e1269e9b95277b7"}
Jul 03 20:47:53.188656 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.188544    2553 scope.go:117] "RemoveContainer" containerID="ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220"
Jul 03 20:47:53.190324 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.190298    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw" event={"ID":"a5daf2a5-3760-4a24-8f48-29898e1368cb","Type":"ContainerStarted","Data":"fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e"}
Jul 03 20:47:53.190445 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.190344    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw" event={"ID":"a5daf2a5-3760-4a24-8f48-29898e1368cb","Type":"ContainerStarted","Data":"96cfd3d3d14e17024fc3892c1a855c7c549f9a41cd71309bb6c3c52a181cc0a4"}
Jul 03 20:47:53.190445 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.190379    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:47:53.197748 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.197693    2553 scope.go:117] "RemoveContainer" containerID="ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220"
Jul 03 20:47:53.198096 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:47:53.198024    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220\": container with ID starting with ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220 not found: ID does not exist" containerID="ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220"
Jul 03 20:47:53.198096 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.198061    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220"} err="failed to get container status \"ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220\": rpc error: code = NotFound desc = could not find container \"ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220\": container with ID starting with ebd00d743a0e910323ab18e0c9a2d5f36fca3fdb53e332c116ce01f041dda220 not found: ID does not exist"
Jul 03 20:47:53.208449 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.208424    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"]
Jul 03 20:47:53.212149 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.212129    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fgmjh"]
Jul 03 20:47:53.232829 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:53.232783    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw" podStartSLOduration=1.232767549 podStartE2EDuration="1.232767549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:47:53.231969012 +0000 UTC m=+645.312710417" watchObservedRunningTime="2026-07-03 20:47:53.232767549 +0000 UTC m=+645.313508957"
Jul 03 20:47:54.870889 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:47:54.870856    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" path="/var/lib/kubelet/pods/fe6af1bd-25bf-4934-a4da-c9e3d2f943ec/volumes"
Jul 03 20:48:04.198210 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:04.198179    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:48:08.106954 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.106923    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"]
Jul 03 20:48:08.107400 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.107314    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" containerName="manager"
Jul 03 20:48:08.107400 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.107332    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" containerName="manager"
Jul 03 20:48:08.107476 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.107413    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6af1bd-25bf-4934-a4da-c9e3d2f943ec" containerName="manager"
Jul 03 20:48:08.110654 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.110631    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.113572 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.113552    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v8nmr\""
Jul 03 20:48:08.121846 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.121822    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"]
Jul 03 20:48:08.140579 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.140551    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:08.213248 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213221    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhxz\" (UniqueName: \"kubernetes.io/projected/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-kube-api-access-nxhxz\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213262    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213293    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213309    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213395    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213448    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213495    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213515    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.213707 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.213534    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314603 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314572    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhxz\" (UniqueName: \"kubernetes.io/projected/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-kube-api-access-nxhxz\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314603 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314607    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314631    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314653    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314694    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314731    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314767    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.314811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314792    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.315163 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.314814    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.315163 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.315046    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.315163 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.315090    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.315163 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.315114    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.315300 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.315217    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.315456 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.315438    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.317027 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.317006    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.317333 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.317313    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.323001 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.322978    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhxz\" (UniqueName: \"kubernetes.io/projected/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-kube-api-access-nxhxz\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.323101 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.322976    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c63b7a3-5b6c-4e4f-9551-1af0d38dd772-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-6jcdd\" (UID: \"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.424970 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.424942    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v8nmr\""
Jul 03 20:48:08.432320 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.432296    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:08.502442 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.502376    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:08.575331 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.575305    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"]
Jul 03 20:48:08.577918 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:48:08.577862    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c63b7a3_5b6c_4e4f_9551_1af0d38dd772.slice/crio-849a19dad03a9fa61853575ee354c289a32782fcce3a229df01738ff79fc6b14 WatchSource:0}: Error finding container 849a19dad03a9fa61853575ee354c289a32782fcce3a229df01738ff79fc6b14: Status 404 returned error can't find the container with id 849a19dad03a9fa61853575ee354c289a32782fcce3a229df01738ff79fc6b14
Jul 03 20:48:08.580197 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.580166    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:48:08.580275 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.580234    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:48:08.580275 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:08.580268    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:48:09.262876 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:09.262842    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" event={"ID":"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772","Type":"ContainerStarted","Data":"9abda6e641599ead4a1ac31999afa0b16263d80848cf879ce349c7de6cb38ab6"}
Jul 03 20:48:09.263255 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:09.262914    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" event={"ID":"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772","Type":"ContainerStarted","Data":"849a19dad03a9fa61853575ee354c289a32782fcce3a229df01738ff79fc6b14"}
Jul 03 20:48:09.286077 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:09.286028    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" podStartSLOduration=1.286012864 podStartE2EDuration="1.286012864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:48:09.283531769 +0000 UTC m=+661.364273177" watchObservedRunningTime="2026-07-03 20:48:09.286012864 +0000 UTC m=+661.366754272"
Jul 03 20:48:09.432651 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:09.432618    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:09.437291 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:09.437270    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:10.266955 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:10.266922    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:10.268253 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:10.268230    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:48:12.954794 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:12.954759    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:13.053860 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.053826    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:13.151850 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.151814    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:13.714563 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.714536    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-djcmr"]
Jul 03 20:48:13.718537 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.718512    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:13.721228 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.721197    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m5gtv\""
Jul 03 20:48:13.728279 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.728258    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-djcmr"]
Jul 03 20:48:13.741533 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.741511    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:13.762963 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.762938    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ztf\" (UniqueName: \"kubernetes.io/projected/1791d65c-105d-4843-aed0-6845c4b63ab6-kube-api-access-v8ztf\") pod \"authorino-58cd955675-djcmr\" (UID: \"1791d65c-105d-4843-aed0-6845c4b63ab6\") " pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:13.802562 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.802508    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:13.863612 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.863574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ztf\" (UniqueName: \"kubernetes.io/projected/1791d65c-105d-4843-aed0-6845c4b63ab6-kube-api-access-v8ztf\") pod \"authorino-58cd955675-djcmr\" (UID: \"1791d65c-105d-4843-aed0-6845c4b63ab6\") " pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:13.872628 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:13.872605    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ztf\" (UniqueName: \"kubernetes.io/projected/1791d65c-105d-4843-aed0-6845c4b63ab6-kube-api-access-v8ztf\") pod \"authorino-58cd955675-djcmr\" (UID: \"1791d65c-105d-4843-aed0-6845c4b63ab6\") " pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:14.029721 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:14.029688    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:14.181989 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:14.181968    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-djcmr"]
Jul 03 20:48:14.184117 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:48:14.184094    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1791d65c_105d_4843_aed0_6845c4b63ab6.slice/crio-cdadc4b3672442edacacf7a56982b04932f3085edbbec5f2ef9096d3fab60d55 WatchSource:0}: Error finding container cdadc4b3672442edacacf7a56982b04932f3085edbbec5f2ef9096d3fab60d55: Status 404 returned error can't find the container with id cdadc4b3672442edacacf7a56982b04932f3085edbbec5f2ef9096d3fab60d55
Jul 03 20:48:14.287218 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:14.287144    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djcmr" event={"ID":"1791d65c-105d-4843-aed0-6845c4b63ab6","Type":"ContainerStarted","Data":"cdadc4b3672442edacacf7a56982b04932f3085edbbec5f2ef9096d3fab60d55"}
Jul 03 20:48:17.304176 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:17.304145    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djcmr" event={"ID":"1791d65c-105d-4843-aed0-6845c4b63ab6","Type":"ContainerStarted","Data":"74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502"}
Jul 03 20:48:17.320008 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:17.319962    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-djcmr" podStartSLOduration=1.259955425 podStartE2EDuration="4.319945679s" totalImagesPullingTime="3.059990254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:48:17.31845333 +0000 UTC m=+669.399194739" watchObservedRunningTime="2026-07-03 20:48:17.319945679 +0000 UTC m=+669.400687087"
Jul 03 20:48:18.997222 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:18.997161    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-djcmr"]
Jul 03 20:48:19.312622 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:19.312582    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-djcmr" podUID="1791d65c-105d-4843-aed0-6845c4b63ab6" containerName="authorino" containerID="cri-o://74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502" gracePeriod=30
Jul 03 20:48:19.550779 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:19.550754    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:19.618156 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:19.618084    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8ztf\" (UniqueName: \"kubernetes.io/projected/1791d65c-105d-4843-aed0-6845c4b63ab6-kube-api-access-v8ztf\") pod \"1791d65c-105d-4843-aed0-6845c4b63ab6\" (UID: \"1791d65c-105d-4843-aed0-6845c4b63ab6\") "
Jul 03 20:48:19.620111 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:19.620089    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1791d65c-105d-4843-aed0-6845c4b63ab6-kube-api-access-v8ztf" (OuterVolumeSpecName: "kube-api-access-v8ztf") pod "1791d65c-105d-4843-aed0-6845c4b63ab6" (UID: "1791d65c-105d-4843-aed0-6845c4b63ab6"). InnerVolumeSpecName "kube-api-access-v8ztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:48:19.718944 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:19.718915    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ztf\" (UniqueName: \"kubernetes.io/projected/1791d65c-105d-4843-aed0-6845c4b63ab6-kube-api-access-v8ztf\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:48:20.318508 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.318467    2553 generic.go:356] "Generic (PLEG): container finished" podID="1791d65c-105d-4843-aed0-6845c4b63ab6" containerID="74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502" exitCode=0
Jul 03 20:48:20.318997 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.318514    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-djcmr"
Jul 03 20:48:20.318997 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.318561    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djcmr" event={"ID":"1791d65c-105d-4843-aed0-6845c4b63ab6","Type":"ContainerDied","Data":"74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502"}
Jul 03 20:48:20.318997 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.318616    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-djcmr" event={"ID":"1791d65c-105d-4843-aed0-6845c4b63ab6","Type":"ContainerDied","Data":"cdadc4b3672442edacacf7a56982b04932f3085edbbec5f2ef9096d3fab60d55"}
Jul 03 20:48:20.318997 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.318643    2553 scope.go:117] "RemoveContainer" containerID="74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502"
Jul 03 20:48:20.328800 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.328118    2553 scope.go:117] "RemoveContainer" containerID="74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502"
Jul 03 20:48:20.328800 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:48:20.328482    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502\": container with ID starting with 74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502 not found: ID does not exist" containerID="74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502"
Jul 03 20:48:20.328800 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.328523    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502"} err="failed to get container status \"74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502\": rpc error: code = NotFound desc = could not find container \"74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502\": container with ID starting with 74e5d7d3c508f817f8f1217256a399fe5281619a76d9cbedd29e32c018901502 not found: ID does not exist"
Jul 03 20:48:20.345422 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.345395    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-djcmr"]
Jul 03 20:48:20.348933 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.348912    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-djcmr"]
Jul 03 20:48:20.870825 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:20.870787    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1791d65c-105d-4843-aed0-6845c4b63ab6" path="/var/lib/kubelet/pods/1791d65c-105d-4843-aed0-6845c4b63ab6/volumes"
Jul 03 20:48:31.786615 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:31.786582    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:44.353705 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.353664    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:44.573641 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.573604    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g8m2f"]
Jul 03 20:48:44.573982 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.573968    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1791d65c-105d-4843-aed0-6845c4b63ab6" containerName="authorino"
Jul 03 20:48:44.574028 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.573984    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1791d65c-105d-4843-aed0-6845c4b63ab6" containerName="authorino"
Jul 03 20:48:44.574091 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.574036    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1791d65c-105d-4843-aed0-6845c4b63ab6" containerName="authorino"
Jul 03 20:48:44.578192 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.578172    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:44.581085 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.581061    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m5gtv\""
Jul 03 20:48:44.581746 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.581723    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g8m2f"]
Jul 03 20:48:44.739828 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.739745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5gpd\" (UniqueName: \"kubernetes.io/projected/2ec7678e-afd1-4666-9402-a32fc9f4212f-kube-api-access-d5gpd\") pod \"authorino-5d4b67c446-g8m2f\" (UID: \"2ec7678e-afd1-4666-9402-a32fc9f4212f\") " pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:44.818069 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.818037    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g8m2f"]
Jul 03 20:48:44.818306 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:48:44.818286    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-d5gpd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-g8m2f" podUID="2ec7678e-afd1-4666-9402-a32fc9f4212f"
Jul 03 20:48:44.825104 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.825078    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:44.840189 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.840166    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5gpd\" (UniqueName: \"kubernetes.io/projected/2ec7678e-afd1-4666-9402-a32fc9f4212f-kube-api-access-d5gpd\") pod \"authorino-5d4b67c446-g8m2f\" (UID: \"2ec7678e-afd1-4666-9402-a32fc9f4212f\") " pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:44.843696 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.843672    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b47b9d9c-xwbh9"]
Jul 03 20:48:44.845959 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.845941    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:44.851887 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.851866    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b47b9d9c-xwbh9"]
Jul 03 20:48:44.853935 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.853912    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5gpd\" (UniqueName: \"kubernetes.io/projected/2ec7678e-afd1-4666-9402-a32fc9f4212f-kube-api-access-d5gpd\") pod \"authorino-5d4b67c446-g8m2f\" (UID: \"2ec7678e-afd1-4666-9402-a32fc9f4212f\") " pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:44.940599 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:44.940569    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnpmx\" (UniqueName: \"kubernetes.io/projected/71e7772a-1855-44e0-b5d0-e98218abeac2-kube-api-access-gnpmx\") pod \"authorino-b47b9d9c-xwbh9\" (UID: \"71e7772a-1855-44e0-b5d0-e98218abeac2\") " pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:45.041465 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.041428    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnpmx\" (UniqueName: \"kubernetes.io/projected/71e7772a-1855-44e0-b5d0-e98218abeac2-kube-api-access-gnpmx\") pod \"authorino-b47b9d9c-xwbh9\" (UID: \"71e7772a-1855-44e0-b5d0-e98218abeac2\") " pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:45.049582 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.049553    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnpmx\" (UniqueName: \"kubernetes.io/projected/71e7772a-1855-44e0-b5d0-e98218abeac2-kube-api-access-gnpmx\") pod \"authorino-b47b9d9c-xwbh9\" (UID: \"71e7772a-1855-44e0-b5d0-e98218abeac2\") " pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:45.094860 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.094822    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b47b9d9c-xwbh9"]
Jul 03 20:48:45.095127 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.095110    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:45.126168 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.126143    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66bccf6785-k7ckv"]
Jul 03 20:48:45.128607 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.128591    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.131320 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.131300    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 03 20:48:45.138303 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.138072    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66bccf6785-k7ckv"]
Jul 03 20:48:45.232246 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.232187    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b47b9d9c-xwbh9"]
Jul 03 20:48:45.234992 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:48:45.234966    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e7772a_1855_44e0_b5d0_e98218abeac2.slice/crio-112a179f4cd234b1a435e9f0783ce08256c36bc245937b30e377f80ae8c0800d WatchSource:0}: Error finding container 112a179f4cd234b1a435e9f0783ce08256c36bc245937b30e377f80ae8c0800d: Status 404 returned error can't find the container with id 112a179f4cd234b1a435e9f0783ce08256c36bc245937b30e377f80ae8c0800d
Jul 03 20:48:45.242779 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.242756    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxnw\" (UniqueName: \"kubernetes.io/projected/89cbd70b-c169-438c-9df9-6f7e98514635-kube-api-access-4rxnw\") pod \"authorino-66bccf6785-k7ckv\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") " pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.242871 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.242835    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89cbd70b-c169-438c-9df9-6f7e98514635-tls-cert\") pod \"authorino-66bccf6785-k7ckv\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") " pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.344133 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.344049    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89cbd70b-c169-438c-9df9-6f7e98514635-tls-cert\") pod \"authorino-66bccf6785-k7ckv\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") " pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.344133 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.344107    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxnw\" (UniqueName: \"kubernetes.io/projected/89cbd70b-c169-438c-9df9-6f7e98514635-kube-api-access-4rxnw\") pod \"authorino-66bccf6785-k7ckv\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") " pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.346525 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.346498    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89cbd70b-c169-438c-9df9-6f7e98514635-tls-cert\") pod \"authorino-66bccf6785-k7ckv\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") " pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.352190 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.352157    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxnw\" (UniqueName: \"kubernetes.io/projected/89cbd70b-c169-438c-9df9-6f7e98514635-kube-api-access-4rxnw\") pod \"authorino-66bccf6785-k7ckv\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") " pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.428797 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.428768    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:45.428797 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.428772    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b47b9d9c-xwbh9" event={"ID":"71e7772a-1855-44e0-b5d0-e98218abeac2","Type":"ContainerStarted","Data":"112a179f4cd234b1a435e9f0783ce08256c36bc245937b30e377f80ae8c0800d"}
Jul 03 20:48:45.434721 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.434701    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:45.439751 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.439735    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:48:45.545382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.545349    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5gpd\" (UniqueName: \"kubernetes.io/projected/2ec7678e-afd1-4666-9402-a32fc9f4212f-kube-api-access-d5gpd\") pod \"2ec7678e-afd1-4666-9402-a32fc9f4212f\" (UID: \"2ec7678e-afd1-4666-9402-a32fc9f4212f\") "
Jul 03 20:48:45.547452 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.547426    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec7678e-afd1-4666-9402-a32fc9f4212f-kube-api-access-d5gpd" (OuterVolumeSpecName: "kube-api-access-d5gpd") pod "2ec7678e-afd1-4666-9402-a32fc9f4212f" (UID: "2ec7678e-afd1-4666-9402-a32fc9f4212f"). InnerVolumeSpecName "kube-api-access-d5gpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:48:45.570619 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.570595    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66bccf6785-k7ckv"]
Jul 03 20:48:45.574207 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:48:45.574186    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89cbd70b_c169_438c_9df9_6f7e98514635.slice/crio-e46ae6c8d364b2c93abc5eef86453d9bf73def8fa0eca5d2f4e6c08373c94b2e WatchSource:0}: Error finding container e46ae6c8d364b2c93abc5eef86453d9bf73def8fa0eca5d2f4e6c08373c94b2e: Status 404 returned error can't find the container with id e46ae6c8d364b2c93abc5eef86453d9bf73def8fa0eca5d2f4e6c08373c94b2e
Jul 03 20:48:45.646308 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:45.646281    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5gpd\" (UniqueName: \"kubernetes.io/projected/2ec7678e-afd1-4666-9402-a32fc9f4212f-kube-api-access-d5gpd\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:48:46.434811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.434773    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bccf6785-k7ckv" event={"ID":"89cbd70b-c169-438c-9df9-6f7e98514635","Type":"ContainerStarted","Data":"bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f"}
Jul 03 20:48:46.434811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.434818    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bccf6785-k7ckv" event={"ID":"89cbd70b-c169-438c-9df9-6f7e98514635","Type":"ContainerStarted","Data":"e46ae6c8d364b2c93abc5eef86453d9bf73def8fa0eca5d2f4e6c08373c94b2e"}
Jul 03 20:48:46.436236 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.436217    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-g8m2f"
Jul 03 20:48:46.436343 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.436262    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b47b9d9c-xwbh9" podUID="71e7772a-1855-44e0-b5d0-e98218abeac2" containerName="authorino" containerID="cri-o://1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03" gracePeriod=30
Jul 03 20:48:46.436343 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.436215    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b47b9d9c-xwbh9" event={"ID":"71e7772a-1855-44e0-b5d0-e98218abeac2","Type":"ContainerStarted","Data":"1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03"}
Jul 03 20:48:46.451310 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.451271    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66bccf6785-k7ckv" podStartSLOduration=1.14038776 podStartE2EDuration="1.451257696s" totalImagesPullingTime="310.869936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:48:46.44985453 +0000 UTC m=+698.530595936" watchObservedRunningTime="2026-07-03 20:48:46.451257696 +0000 UTC m=+698.531999105"
Jul 03 20:48:46.469595 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.469540    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b47b9d9c-xwbh9" podStartSLOduration=2.139798545 podStartE2EDuration="2.469521683s" totalImagesPullingTime="329.723138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:48:46.46781201 +0000 UTC m=+698.548553421" watchObservedRunningTime="2026-07-03 20:48:46.469521683 +0000 UTC m=+698.550263094"
Jul 03 20:48:46.497056 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.497025    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g8m2f"]
Jul 03 20:48:46.498619 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.498590    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-g8m2f"]
Jul 03 20:48:46.676808 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.676783    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:46.857494 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.857463    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnpmx\" (UniqueName: \"kubernetes.io/projected/71e7772a-1855-44e0-b5d0-e98218abeac2-kube-api-access-gnpmx\") pod \"71e7772a-1855-44e0-b5d0-e98218abeac2\" (UID: \"71e7772a-1855-44e0-b5d0-e98218abeac2\") "
Jul 03 20:48:46.859527 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.859504    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e7772a-1855-44e0-b5d0-e98218abeac2-kube-api-access-gnpmx" (OuterVolumeSpecName: "kube-api-access-gnpmx") pod "71e7772a-1855-44e0-b5d0-e98218abeac2" (UID: "71e7772a-1855-44e0-b5d0-e98218abeac2"). InnerVolumeSpecName "kube-api-access-gnpmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:48:46.870641 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.870617    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec7678e-afd1-4666-9402-a32fc9f4212f" path="/var/lib/kubelet/pods/2ec7678e-afd1-4666-9402-a32fc9f4212f/volumes"
Jul 03 20:48:46.958310 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:46.958276    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnpmx\" (UniqueName: \"kubernetes.io/projected/71e7772a-1855-44e0-b5d0-e98218abeac2-kube-api-access-gnpmx\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:48:47.441926 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.441876    2553 generic.go:356] "Generic (PLEG): container finished" podID="71e7772a-1855-44e0-b5d0-e98218abeac2" containerID="1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03" exitCode=0
Jul 03 20:48:47.442317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.441936    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b47b9d9c-xwbh9"
Jul 03 20:48:47.442317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.441970    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b47b9d9c-xwbh9" event={"ID":"71e7772a-1855-44e0-b5d0-e98218abeac2","Type":"ContainerDied","Data":"1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03"}
Jul 03 20:48:47.442317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.442017    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b47b9d9c-xwbh9" event={"ID":"71e7772a-1855-44e0-b5d0-e98218abeac2","Type":"ContainerDied","Data":"112a179f4cd234b1a435e9f0783ce08256c36bc245937b30e377f80ae8c0800d"}
Jul 03 20:48:47.442317 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.442040    2553 scope.go:117] "RemoveContainer" containerID="1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03"
Jul 03 20:48:47.450276 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.450248    2553 scope.go:117] "RemoveContainer" containerID="1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03"
Jul 03 20:48:47.450519 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:48:47.450502    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03\": container with ID starting with 1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03 not found: ID does not exist" containerID="1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03"
Jul 03 20:48:47.450571 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.450536    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03"} err="failed to get container status \"1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03\": rpc error: code = NotFound desc = could not find container \"1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03\": container with ID starting with 1a5995594f6eb4c6b0164ee8f8fbd18c450df63112afbd1372db8aed41be5b03 not found: ID does not exist"
Jul 03 20:48:47.458933 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.458913    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b47b9d9c-xwbh9"]
Jul 03 20:48:47.461260 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:47.461242    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b47b9d9c-xwbh9"]
Jul 03 20:48:48.869659 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:48.869617    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e7772a-1855-44e0-b5d0-e98218abeac2" path="/var/lib/kubelet/pods/71e7772a-1855-44e0-b5d0-e98218abeac2/volumes"
Jul 03 20:48:50.849188 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:50.849146    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:50.948815 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:50.948785    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:51.036240 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:51.036209    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:51.129616 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:51.129534    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:53.730160 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:53.730128    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:48:53.828049 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:48:53.828018    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:04.550818 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:04.550771    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:25.004623 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:25.004587    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:25.474475 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:25.474437    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:25.777968 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:25.777935    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:25.999767 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:25.999736    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:26.097862 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:26.097785    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:33.650456 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:33.650366    2553 generic.go:356] "Generic (PLEG): container finished" podID="8c63b7a3-5b6c-4e4f-9551-1af0d38dd772" containerID="9abda6e641599ead4a1ac31999afa0b16263d80848cf879ce349c7de6cb38ab6" exitCode=0
Jul 03 20:49:33.650853 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:33.650451    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" event={"ID":"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772","Type":"ContainerDied","Data":"9abda6e641599ead4a1ac31999afa0b16263d80848cf879ce349c7de6cb38ab6"}
Jul 03 20:49:33.650853 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:33.650847    2553 scope.go:117] "RemoveContainer" containerID="9abda6e641599ead4a1ac31999afa0b16263d80848cf879ce349c7de6cb38ab6"
Jul 03 20:49:33.651835 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:33.651809    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:49:33.651924 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:33.651873    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:49:33.651971 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:33.651924    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 20:49:34.660338 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:34.660302    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" event={"ID":"8c63b7a3-5b6c-4e4f-9551-1af0d38dd772","Type":"ContainerStarted","Data":"b088f4235cf28c9e616a6b4cd003a3d44abbfc955b4d4081f8a4ca1c04f86132"}
Jul 03 20:49:35.299979 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:35.299936    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:35.393251 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:35.393220    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:35.433312 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:35.433275    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:49:35.434427 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:35.434400    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6jcdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 03 20:49:35.434534 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:35.434449    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" podUID="8c63b7a3-5b6c-4e4f-9551-1af0d38dd772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 03 20:49:36.433438 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:36.433407    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6jcdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 03 20:49:36.433873 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:36.433472    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" podUID="8c63b7a3-5b6c-4e4f-9551-1af0d38dd772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 03 20:49:37.432969 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:37.432933    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6jcdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 03 20:49:37.433146 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:37.432991    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" podUID="8c63b7a3-5b6c-4e4f-9551-1af0d38dd772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 03 20:49:38.433702 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:38.433659    2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-6jcdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Jul 03 20:49:38.434108 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:38.433730    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd" podUID="8c63b7a3-5b6c-4e4f-9551-1af0d38dd772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Jul 03 20:49:39.436870 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:39.436841    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:49:39.437266 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:39.437127    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:49:39.437779 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:39.437763    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-6jcdd"
Jul 03 20:49:52.558672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.558629    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"]
Jul 03 20:49:52.559239 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.559218    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e7772a-1855-44e0-b5d0-e98218abeac2" containerName="authorino"
Jul 03 20:49:52.559308 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.559242    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e7772a-1855-44e0-b5d0-e98218abeac2" containerName="authorino"
Jul 03 20:49:52.559373 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.559352    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e7772a-1855-44e0-b5d0-e98218abeac2" containerName="authorino"
Jul 03 20:49:52.561433 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.561413    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.564124 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.564100    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 03 20:49:52.564252 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.564102    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 03 20:49:52.564312 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.564258    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qjt4q\""
Jul 03 20:49:52.564369 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.564349    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 03 20:49:52.570476 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.570284    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"]
Jul 03 20:49:52.590298 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.590272    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:52.594432 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.594410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ed86e7-226a-4386-97eb-c64e7f4870e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.594534 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.594445    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lzzs\" (UniqueName: \"kubernetes.io/projected/e6ed86e7-226a-4386-97eb-c64e7f4870e1-kube-api-access-9lzzs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.594534 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.594470    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.594607 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.594534    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.594607 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.594562    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.594607 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.594583    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696063 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696020    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lzzs\" (UniqueName: \"kubernetes.io/projected/e6ed86e7-226a-4386-97eb-c64e7f4870e1-kube-api-access-9lzzs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696086    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696125    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696155    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696187    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696420 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696237    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ed86e7-226a-4386-97eb-c64e7f4870e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696574 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696551    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696645 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696612    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.696696 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.696645    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.698426 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.698401    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6ed86e7-226a-4386-97eb-c64e7f4870e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.698785 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.698767    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6ed86e7-226a-4386-97eb-c64e7f4870e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.704680 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.704651    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lzzs\" (UniqueName: \"kubernetes.io/projected/e6ed86e7-226a-4386-97eb-c64e7f4870e1-kube-api-access-9lzzs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h\" (UID: \"e6ed86e7-226a-4386-97eb-c64e7f4870e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:52.874811 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:52.874734    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:49:53.011056 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:53.011008    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"]
Jul 03 20:49:53.013387 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:49:53.013359    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ed86e7_226a_4386_97eb_c64e7f4870e1.slice/crio-a3a05b571292ba755a16f32ea84fa34937a17412ff5a3ff01b6e7d0951d2965c WatchSource:0}: Error finding container a3a05b571292ba755a16f32ea84fa34937a17412ff5a3ff01b6e7d0951d2965c: Status 404 returned error can't find the container with id a3a05b571292ba755a16f32ea84fa34937a17412ff5a3ff01b6e7d0951d2965c
Jul 03 20:49:53.752612 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:53.752573    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h" event={"ID":"e6ed86e7-226a-4386-97eb-c64e7f4870e1","Type":"ContainerStarted","Data":"a3a05b571292ba755a16f32ea84fa34937a17412ff5a3ff01b6e7d0951d2965c"}
Jul 03 20:49:54.426046 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:54.426015    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:54.487681 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:54.487650    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:49:54.758119 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:54.758080    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h" event={"ID":"e6ed86e7-226a-4386-97eb-c64e7f4870e1","Type":"ContainerStarted","Data":"2d7edc6295836b644b8955f87ec54e538f500513d287c447a8403414cbcebba6"}
Jul 03 20:49:54.776164 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:49:54.776104    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h" podStartSLOduration=1.5304987909999999 podStartE2EDuration="2.77608926s" totalImagesPullingTime="1.245590469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:49:54.774430235 +0000 UTC m=+766.855171684" watchObservedRunningTime="2026-07-03 20:49:54.77608926 +0000 UTC m=+766.856830667"
Jul 03 20:50:02.875483 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:02.875439    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:50:02.876086 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:02.875527    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:50:02.888244 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:02.888221    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:50:03.811199 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:03.811168    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h"
Jul 03 20:50:07.469345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:07.469311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:15.689945 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:15.689907    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:23.290795 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:23.290754    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:35.282459 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:35.282377    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:35.332501 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:35.332469    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:37.002862 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:37.002829    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:37.103203 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:37.103174    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:47.252475 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.252444    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"]
Jul 03 20:50:47.256206 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.256182    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.259327 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.259307    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 03 20:50:47.265841 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.265819    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"]
Jul 03 20:50:47.280265 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.280234    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.280403 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.280274    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllfw\" (UniqueName: \"kubernetes.io/projected/7b539ac1-8323-42e1-a854-403e03cbb65b-kube-api-access-fllfw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.280403 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.280300    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b539ac1-8323-42e1-a854-403e03cbb65b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.280516 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.280463    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.280516 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.280511    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.280615 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.280532    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.298133 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.298092    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:47.381076 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381045    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381076 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381080    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381296 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381097    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381296 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381134    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381296 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381167    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllfw\" (UniqueName: \"kubernetes.io/projected/7b539ac1-8323-42e1-a854-403e03cbb65b-kube-api-access-fllfw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381296 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381210    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b539ac1-8323-42e1-a854-403e03cbb65b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381525 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381495    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381570    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.381672 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.381667    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.383466 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.383446    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b539ac1-8323-42e1-a854-403e03cbb65b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.383634 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.383616    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b539ac1-8323-42e1-a854-403e03cbb65b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.388957 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.388937    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllfw\" (UniqueName: \"kubernetes.io/projected/7b539ac1-8323-42e1-a854-403e03cbb65b-kube-api-access-fllfw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hqk6t\" (UID: \"7b539ac1-8323-42e1-a854-403e03cbb65b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.568631 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.568601    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:47.702987 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:47.702963    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"]
Jul 03 20:50:47.705505 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:50:47.705478    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b539ac1_8323_42e1_a854_403e03cbb65b.slice/crio-61c4f58eebd418baab6f622222dc88347854603f027050fbc4f4bbee88347777 WatchSource:0}: Error finding container 61c4f58eebd418baab6f622222dc88347854603f027050fbc4f4bbee88347777: Status 404 returned error can't find the container with id 61c4f58eebd418baab6f622222dc88347854603f027050fbc4f4bbee88347777
Jul 03 20:50:48.008394 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:48.008358    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t" event={"ID":"7b539ac1-8323-42e1-a854-403e03cbb65b","Type":"ContainerStarted","Data":"da03d29b77b079553fb8976d85ae813c53dce3762bec17a7d7bea8489c934481"}
Jul 03 20:50:48.008560 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:48.008457    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t" event={"ID":"7b539ac1-8323-42e1-a854-403e03cbb65b","Type":"ContainerStarted","Data":"61c4f58eebd418baab6f622222dc88347854603f027050fbc4f4bbee88347777"}
Jul 03 20:50:48.028837 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:48.028791    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t" podStartSLOduration=0.857222105 podStartE2EDuration="1.028777143s" totalImagesPullingTime="171.555038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:50:48.026847322 +0000 UTC m=+820.107588730" watchObservedRunningTime="2026-07-03 20:50:48.028777143 +0000 UTC m=+820.109518551"
Jul 03 20:50:52.576945 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:52.576872    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:56.483149 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:56.483080    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:50:57.568932 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:57.568881    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:57.569308 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:57.568960    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:57.581205 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:57.581176    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:50:58.068838 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:50:58.068811    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hqk6t"
Jul 03 20:51:20.383887 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:20.383844    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:51:24.930126 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:24.930092    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f754c655b-k6lnj"]
Jul 03 20:51:24.933711 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:24.933690    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:24.940814 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:24.940501    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f754c655b-k6lnj"]
Jul 03 20:51:25.018392 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.018363    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc970611-5804-4465-b9ed-3747c018a172-tls-cert\") pod \"authorino-6f754c655b-k6lnj\" (UID: \"bc970611-5804-4465-b9ed-3747c018a172\") " pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.018558 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.018480    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsx7\" (UniqueName: \"kubernetes.io/projected/bc970611-5804-4465-b9ed-3747c018a172-kube-api-access-dcsx7\") pod \"authorino-6f754c655b-k6lnj\" (UID: \"bc970611-5804-4465-b9ed-3747c018a172\") " pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.119117 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.119075    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsx7\" (UniqueName: \"kubernetes.io/projected/bc970611-5804-4465-b9ed-3747c018a172-kube-api-access-dcsx7\") pod \"authorino-6f754c655b-k6lnj\" (UID: \"bc970611-5804-4465-b9ed-3747c018a172\") " pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.119284 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.119127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc970611-5804-4465-b9ed-3747c018a172-tls-cert\") pod \"authorino-6f754c655b-k6lnj\" (UID: \"bc970611-5804-4465-b9ed-3747c018a172\") " pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.121499 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.121479    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bc970611-5804-4465-b9ed-3747c018a172-tls-cert\") pod \"authorino-6f754c655b-k6lnj\" (UID: \"bc970611-5804-4465-b9ed-3747c018a172\") " pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.129935 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.129883    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsx7\" (UniqueName: \"kubernetes.io/projected/bc970611-5804-4465-b9ed-3747c018a172-kube-api-access-dcsx7\") pod \"authorino-6f754c655b-k6lnj\" (UID: \"bc970611-5804-4465-b9ed-3747c018a172\") " pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.243806 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.243733    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f754c655b-k6lnj"
Jul 03 20:51:25.387856 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.387826    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f754c655b-k6lnj"]
Jul 03 20:51:25.389976 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:51:25.389939    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc970611_5804_4465_b9ed_3747c018a172.slice/crio-bcd6f45ffb5c857e67c78f79b10f19f39a9ae47d555c54c9aa202b68c8c77f50 WatchSource:0}: Error finding container bcd6f45ffb5c857e67c78f79b10f19f39a9ae47d555c54c9aa202b68c8c77f50: Status 404 returned error can't find the container with id bcd6f45ffb5c857e67c78f79b10f19f39a9ae47d555c54c9aa202b68c8c77f50
Jul 03 20:51:25.393705 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.393673    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:51:25.492629 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:25.492595    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:51:26.189796 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.189696    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f754c655b-k6lnj" event={"ID":"bc970611-5804-4465-b9ed-3747c018a172","Type":"ContainerStarted","Data":"f7178f045dcde47ec3c561cc64b833638c9d843f23391e1f7913d5009a157d8b"}
Jul 03 20:51:26.189796 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.189749    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f754c655b-k6lnj" event={"ID":"bc970611-5804-4465-b9ed-3747c018a172","Type":"ContainerStarted","Data":"bcd6f45ffb5c857e67c78f79b10f19f39a9ae47d555c54c9aa202b68c8c77f50"}
Jul 03 20:51:26.206362 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.206302    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f754c655b-k6lnj" podStartSLOduration=1.721698454 podStartE2EDuration="2.206282426s" totalImagesPullingTime="484.583972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:51:26.206103266 +0000 UTC m=+858.286844676" watchObservedRunningTime="2026-07-03 20:51:26.206282426 +0000 UTC m=+858.287023837"
Jul 03 20:51:26.236147 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.235988    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66bccf6785-k7ckv"]
Jul 03 20:51:26.236331 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.236186    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-66bccf6785-k7ckv" podUID="89cbd70b-c169-438c-9df9-6f7e98514635" containerName="authorino" containerID="cri-o://bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f" gracePeriod=30
Jul 03 20:51:26.512678 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.512646    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:51:26.580593 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.580540    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:51:26.633461 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.633423    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89cbd70b-c169-438c-9df9-6f7e98514635-tls-cert\") pod \"89cbd70b-c169-438c-9df9-6f7e98514635\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") "
Jul 03 20:51:26.633647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.633518    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rxnw\" (UniqueName: \"kubernetes.io/projected/89cbd70b-c169-438c-9df9-6f7e98514635-kube-api-access-4rxnw\") pod \"89cbd70b-c169-438c-9df9-6f7e98514635\" (UID: \"89cbd70b-c169-438c-9df9-6f7e98514635\") "
Jul 03 20:51:26.635881 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.635800    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cbd70b-c169-438c-9df9-6f7e98514635-kube-api-access-4rxnw" (OuterVolumeSpecName: "kube-api-access-4rxnw") pod "89cbd70b-c169-438c-9df9-6f7e98514635" (UID: "89cbd70b-c169-438c-9df9-6f7e98514635"). InnerVolumeSpecName "kube-api-access-4rxnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:51:26.645599 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.645567    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cbd70b-c169-438c-9df9-6f7e98514635-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "89cbd70b-c169-438c-9df9-6f7e98514635" (UID: "89cbd70b-c169-438c-9df9-6f7e98514635"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 20:51:26.675331 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.675289    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:51:26.735165 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.735128    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxnw\" (UniqueName: \"kubernetes.io/projected/89cbd70b-c169-438c-9df9-6f7e98514635-kube-api-access-4rxnw\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:51:26.735165 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:26.735157    2553 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89cbd70b-c169-438c-9df9-6f7e98514635-tls-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:51:27.197211 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.197100    2553 generic.go:356] "Generic (PLEG): container finished" podID="89cbd70b-c169-438c-9df9-6f7e98514635" containerID="bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f" exitCode=0
Jul 03 20:51:27.197676 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.197209    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bccf6785-k7ckv"
Jul 03 20:51:27.197676 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.197279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bccf6785-k7ckv" event={"ID":"89cbd70b-c169-438c-9df9-6f7e98514635","Type":"ContainerDied","Data":"bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f"}
Jul 03 20:51:27.197676 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.197315    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bccf6785-k7ckv" event={"ID":"89cbd70b-c169-438c-9df9-6f7e98514635","Type":"ContainerDied","Data":"e46ae6c8d364b2c93abc5eef86453d9bf73def8fa0eca5d2f4e6c08373c94b2e"}
Jul 03 20:51:27.197676 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.197338    2553 scope.go:117] "RemoveContainer" containerID="bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f"
Jul 03 20:51:27.207380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.207356    2553 scope.go:117] "RemoveContainer" containerID="bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f"
Jul 03 20:51:27.207688 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:51:27.207660    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f\": container with ID starting with bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f not found: ID does not exist" containerID="bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f"
Jul 03 20:51:27.207746 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.207689    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f"} err="failed to get container status \"bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f\": rpc error: code = NotFound desc = could not find container \"bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f\": container with ID starting with bfa2b3d45e1dbc0cb93a0e087dca4ebc948f6930c24c149fbc2efb1b2bb4fe5f not found: ID does not exist"
Jul 03 20:51:27.215109 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.215084    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66bccf6785-k7ckv"]
Jul 03 20:51:27.220288 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:27.220260    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66bccf6785-k7ckv"]
Jul 03 20:51:28.871834 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:51:28.871800    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89cbd70b-c169-438c-9df9-6f7e98514635" path="/var/lib/kubelet/pods/89cbd70b-c169-438c-9df9-6f7e98514635/volumes"
Jul 03 20:52:10.750967 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:52:10.750928    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:52:11.927616 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:52:11.927584    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:52:36.780011 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:52:36.779979    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:52:37.948621 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:52:37.948588    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:15.563380 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:15.563351    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:15.655769 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:15.655735    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:15.769655 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:15.769620    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:21.438412 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:21.438383    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:21.543090 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:21.543054    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:21.639581 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:21.639553    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:21.834345 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:21.834314    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:27.403342 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:27.403308    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:53:27.468393 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:53:27.468358    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:57:18.383091 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:18.383057    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"]
Jul 03 20:57:18.383647 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:18.383277    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw" podUID="a5daf2a5-3760-4a24-8f48-29898e1368cb" containerName="manager" containerID="cri-o://fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e" gracePeriod=10
Jul 03 20:57:21.534796 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.534769    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:57:21.639104 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.639020    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gszlf\" (UniqueName: \"kubernetes.io/projected/a5daf2a5-3760-4a24-8f48-29898e1368cb-kube-api-access-gszlf\") pod \"a5daf2a5-3760-4a24-8f48-29898e1368cb\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") "
Jul 03 20:57:21.639104 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.639077    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5daf2a5-3760-4a24-8f48-29898e1368cb-extensions-socket-volume\") pod \"a5daf2a5-3760-4a24-8f48-29898e1368cb\" (UID: \"a5daf2a5-3760-4a24-8f48-29898e1368cb\") "
Jul 03 20:57:21.639464 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.639441    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5daf2a5-3760-4a24-8f48-29898e1368cb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a5daf2a5-3760-4a24-8f48-29898e1368cb" (UID: "a5daf2a5-3760-4a24-8f48-29898e1368cb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 20:57:21.641233 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.641197    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5daf2a5-3760-4a24-8f48-29898e1368cb-kube-api-access-gszlf" (OuterVolumeSpecName: "kube-api-access-gszlf") pod "a5daf2a5-3760-4a24-8f48-29898e1368cb" (UID: "a5daf2a5-3760-4a24-8f48-29898e1368cb"). InnerVolumeSpecName "kube-api-access-gszlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 20:57:21.740083 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.740048    2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5daf2a5-3760-4a24-8f48-29898e1368cb-extensions-socket-volume\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:57:21.740083 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.740075    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszlf\" (UniqueName: \"kubernetes.io/projected/a5daf2a5-3760-4a24-8f48-29898e1368cb-kube-api-access-gszlf\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 20:57:21.842816 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.842784    2553 generic.go:356] "Generic (PLEG): container finished" podID="a5daf2a5-3760-4a24-8f48-29898e1368cb" containerID="fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e" exitCode=0
Jul 03 20:57:21.842987 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.842832    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"
Jul 03 20:57:21.842987 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.842871    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw" event={"ID":"a5daf2a5-3760-4a24-8f48-29898e1368cb","Type":"ContainerDied","Data":"fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e"}
Jul 03 20:57:21.842987 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.842964    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw" event={"ID":"a5daf2a5-3760-4a24-8f48-29898e1368cb","Type":"ContainerDied","Data":"96cfd3d3d14e17024fc3892c1a855c7c549f9a41cd71309bb6c3c52a181cc0a4"}
Jul 03 20:57:21.842987 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.842986    2553 scope.go:117] "RemoveContainer" containerID="fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e"
Jul 03 20:57:21.852650 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.852635    2553 scope.go:117] "RemoveContainer" containerID="fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e"
Jul 03 20:57:21.852864 ip-10-0-129-236 kubenswrapper[2553]: E0703 20:57:21.852843    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e\": container with ID starting with fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e not found: ID does not exist" containerID="fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e"
Jul 03 20:57:21.852984 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.852870    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e"} err="failed to get container status \"fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e\": rpc error: code = NotFound desc = could not find container \"fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e\": container with ID starting with fe2a7c791ed82e0e8818ce333779874f08155c874abde3dfa113fb891efc885e not found: ID does not exist"
Jul 03 20:57:21.864378 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.864356    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"]
Jul 03 20:57:21.868010 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:21.867988    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jtwtw"]
Jul 03 20:57:22.870276 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:57:22.870244    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5daf2a5-3760-4a24-8f48-29898e1368cb" path="/var/lib/kubelet/pods/a5daf2a5-3760-4a24-8f48-29898e1368cb/volumes"
Jul 03 20:58:19.514382 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514348    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"]
Jul 03 20:58:19.514781 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514670    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cbd70b-c169-438c-9df9-6f7e98514635" containerName="authorino"
Jul 03 20:58:19.514781 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514680    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cbd70b-c169-438c-9df9-6f7e98514635" containerName="authorino"
Jul 03 20:58:19.514781 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514695    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5daf2a5-3760-4a24-8f48-29898e1368cb" containerName="manager"
Jul 03 20:58:19.514781 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514701    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5daf2a5-3760-4a24-8f48-29898e1368cb" containerName="manager"
Jul 03 20:58:19.514781 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514764    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5daf2a5-3760-4a24-8f48-29898e1368cb" containerName="manager"
Jul 03 20:58:19.514781 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.514772    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cbd70b-c169-438c-9df9-6f7e98514635" containerName="authorino"
Jul 03 20:58:19.518496 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.518478    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.521425 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.521398    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tc8gn\""
Jul 03 20:58:19.528925 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.528905    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"]
Jul 03 20:58:19.638174 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.638133    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wjsj\" (UniqueName: \"kubernetes.io/projected/d0db8825-f5ee-400a-bcff-5930dd67b09b-kube-api-access-8wjsj\") pod \"kuadrant-operator-controller-manager-86878c769f-rz4fr\" (UID: \"d0db8825-f5ee-400a-bcff-5930dd67b09b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.638362 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.638245    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0db8825-f5ee-400a-bcff-5930dd67b09b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rz4fr\" (UID: \"d0db8825-f5ee-400a-bcff-5930dd67b09b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.739287 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.739250    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0db8825-f5ee-400a-bcff-5930dd67b09b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rz4fr\" (UID: \"d0db8825-f5ee-400a-bcff-5930dd67b09b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.739431 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.739328    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wjsj\" (UniqueName: \"kubernetes.io/projected/d0db8825-f5ee-400a-bcff-5930dd67b09b-kube-api-access-8wjsj\") pod \"kuadrant-operator-controller-manager-86878c769f-rz4fr\" (UID: \"d0db8825-f5ee-400a-bcff-5930dd67b09b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.739613 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.739592    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0db8825-f5ee-400a-bcff-5930dd67b09b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rz4fr\" (UID: \"d0db8825-f5ee-400a-bcff-5930dd67b09b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.748859 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.748813    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wjsj\" (UniqueName: \"kubernetes.io/projected/d0db8825-f5ee-400a-bcff-5930dd67b09b-kube-api-access-8wjsj\") pod \"kuadrant-operator-controller-manager-86878c769f-rz4fr\" (UID: \"d0db8825-f5ee-400a-bcff-5930dd67b09b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.828961 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.828930    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:19.967183 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:19.967157    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"]
Jul 03 20:58:19.970251 ip-10-0-129-236 kubenswrapper[2553]: W0703 20:58:19.970222    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0db8825_f5ee_400a_bcff_5930dd67b09b.slice/crio-c1423065ea04c90b029fc1e51d880285977fbf57e6f4b49aeccd4a853da916f8 WatchSource:0}: Error finding container c1423065ea04c90b029fc1e51d880285977fbf57e6f4b49aeccd4a853da916f8: Status 404 returned error can't find the container with id c1423065ea04c90b029fc1e51d880285977fbf57e6f4b49aeccd4a853da916f8
Jul 03 20:58:20.104093 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:20.104014    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr" event={"ID":"d0db8825-f5ee-400a-bcff-5930dd67b09b","Type":"ContainerStarted","Data":"9bfc91a64305af47700acd08f23c39851959227f535aeb7ee1da6aecc37f21f6"}
Jul 03 20:58:20.104093 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:20.104062    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr" event={"ID":"d0db8825-f5ee-400a-bcff-5930dd67b09b","Type":"ContainerStarted","Data":"c1423065ea04c90b029fc1e51d880285977fbf57e6f4b49aeccd4a853da916f8"}
Jul 03 20:58:20.104300 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:20.104155    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:20.152199 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:20.152144    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr" podStartSLOduration=1.152127645 podStartE2EDuration="1.152127645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 20:58:20.148811945 +0000 UTC m=+1272.229553353" watchObservedRunningTime="2026-07-03 20:58:20.152127645 +0000 UTC m=+1272.232869052"
Jul 03 20:58:31.110597 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:31.110565    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rz4fr"
Jul 03 20:58:39.305158 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:58:39.305118    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 20:59:51.844070 ip-10-0-129-236 kubenswrapper[2553]: I0703 20:59:51.843980    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:00:03.100171 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:00:03.100129    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:00:11.472595 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:00:11.472543    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:05.855467 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:05.855387    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:14.686178 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:14.686142    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:22.951213 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:22.951177    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:28.024262 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:28.024231    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:30.840859 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:30.840811    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:30.938631 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:30.938601    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:31.299134 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:31.299094    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:36.956042 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:36.956003    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:37.027584 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.027548    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:37.133216 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.133186    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"]
Jul 03 21:04:37.136869 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.136805    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.139578 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.139558    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-753c62a5-openshift-default-dockercfg-nzl2c\""
Jul 03 21:04:37.163836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.163803    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"]
Jul 03 21:04:37.168485 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.168460    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:37.234366 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.234285    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:37.306814 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.306775    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-socket\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.306825    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-data\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.306842    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfgs\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-kube-api-access-lzfgs\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.306960    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-envoy\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.306989    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-podinfo\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.307024    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-certs\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307261 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.307040    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istiod-ca-cert\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307261 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.307088    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-credential-socket\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.307261 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.307104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-token\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408545 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408511    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-data\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408545 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408545    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfgs\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-kube-api-access-lzfgs\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408744 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408565    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-envoy\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408744 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408705    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-podinfo\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408839 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408760    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-certs\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408839 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408785    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istiod-ca-cert\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.408839 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-credential-socket\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409034 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408854    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-token\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409034 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.408995    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-data\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409034 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.409003    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-socket\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409199 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.409162    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-certs\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409278 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.409253    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-credential-socket\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409466 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.409436    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-socket\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.409563 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.409540    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istiod-ca-cert\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.410870 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.410854    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-envoy\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.411236 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.411220    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-podinfo\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.416393 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.416366    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-token\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.416595 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.416575    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfgs\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-kube-api-access-lzfgs\") pod \"e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") " pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.446642 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.446613    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:37.582762 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.582729    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"]
Jul 03 21:04:37.583934 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:04:37.583887    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8e585c_13e0_4a9d_bb03_69a3ec39e654.slice/crio-e54158b989f688b794eadf516fe5bd34b3a749a438d6073c84fbda2281994fe7 WatchSource:0}: Error finding container e54158b989f688b794eadf516fe5bd34b3a749a438d6073c84fbda2281994fe7: Status 404 returned error can't find the container with id e54158b989f688b794eadf516fe5bd34b3a749a438d6073c84fbda2281994fe7
Jul 03 21:04:37.586009 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.585978    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:04:37.586085 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.586046    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:04:37.586085 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.586074    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:04:37.874955 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.874923    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k" event={"ID":"2d8e585c-13e0-4a9d-bb03-69a3ec39e654","Type":"ContainerStarted","Data":"54c0628b460bb36f7eb03ab970ea22a26727a02a6172d50c9265d767e32dd66a"}
Jul 03 21:04:37.875271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.874974    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k" event={"ID":"2d8e585c-13e0-4a9d-bb03-69a3ec39e654","Type":"ContainerStarted","Data":"e54158b989f688b794eadf516fe5bd34b3a749a438d6073c84fbda2281994fe7"}
Jul 03 21:04:37.899057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:37.898999    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k" podStartSLOduration=0.898975629 podStartE2EDuration="898.975629ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:04:37.895719977 +0000 UTC m=+1649.976461385" watchObservedRunningTime="2026-07-03 21:04:37.898975629 +0000 UTC m=+1649.979717039"
Jul 03 21:04:38.339242 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.339204    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:38.439800 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.439773    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:38.447797 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.447776    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:38.452487 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.452467    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:38.545387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.545360    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:38.727622 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.727535    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:38.880030 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.880000    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:38.881281 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:38.881259    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:39.055375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:39.055347    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:39.159234 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:39.159202    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:41.132211 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:41.132174    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:43.444393 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:43.444360    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:43.604643 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:43.604605    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"]
Jul 03 21:04:43.605152 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:43.605117    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k" podUID="2d8e585c-13e0-4a9d-bb03-69a3ec39e654" containerName="istio-proxy" containerID="cri-o://54c0628b460bb36f7eb03ab970ea22a26727a02a6172d50c9265d767e32dd66a" gracePeriod=30
Jul 03 21:04:43.632762 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:43.632731    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:43.727037 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:43.723346    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:43.822490 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:43.822449    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:45.747352 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:45.747325    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:48.933977 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:48.933949    2553 generic.go:356] "Generic (PLEG): container finished" podID="2d8e585c-13e0-4a9d-bb03-69a3ec39e654" containerID="54c0628b460bb36f7eb03ab970ea22a26727a02a6172d50c9265d767e32dd66a" exitCode=0
Jul 03 21:04:48.934331 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:48.934004    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k" event={"ID":"2d8e585c-13e0-4a9d-bb03-69a3ec39e654","Type":"ContainerDied","Data":"54c0628b460bb36f7eb03ab970ea22a26727a02a6172d50c9265d767e32dd66a"}
Jul 03 21:04:48.934331 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:48.934063    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k" event={"ID":"2d8e585c-13e0-4a9d-bb03-69a3ec39e654","Type":"ContainerDied","Data":"e54158b989f688b794eadf516fe5bd34b3a749a438d6073c84fbda2281994fe7"}
Jul 03 21:04:48.934331 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:48.934095    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e54158b989f688b794eadf516fe5bd34b3a749a438d6073c84fbda2281994fe7"
Jul 03 21:04:48.945178 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:48.945156    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:49.016178 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016089    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-certs\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016178 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016145    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-envoy\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016178 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016174    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-credential-socket\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016237    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istiod-ca-cert\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016274    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-socket\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016319    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzfgs\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-kube-api-access-lzfgs\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016354    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-podinfo\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016381    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-token\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016421    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-data\") pod \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\" (UID: \"2d8e585c-13e0-4a9d-bb03-69a3ec39e654\") "
Jul 03 21:04:49.016767 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016522    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:04:49.016767 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016644    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:04:49.016981 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016792    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:04:49.016981 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016832    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.016981 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.016854    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-credential-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.017218 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.017065    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 21:04:49.017322 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.017285    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:04:49.018662 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.018636    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:04:49.018756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.018658    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 03 21:04:49.018812 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.018762    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:04:49.018850 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.018830    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-kube-api-access-lzfgs" (OuterVolumeSpecName: "kube-api-access-lzfgs") pod "2d8e585c-13e0-4a9d-bb03-69a3ec39e654" (UID: "2d8e585c-13e0-4a9d-bb03-69a3ec39e654"). InnerVolumeSpecName "kube-api-access-lzfgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:04:49.117620 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117591    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-podinfo\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.117620 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117615    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-token\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.117620 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117624    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-data\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.117922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117633    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-workload-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.117922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117641    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istio-envoy\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.117922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117648    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-istiod-ca-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.117922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.117655    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfgs\" (UniqueName: \"kubernetes.io/projected/2d8e585c-13e0-4a9d-bb03-69a3ec39e654-kube-api-access-lzfgs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:04:49.308438 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.308408    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:49.364611 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.364581    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:49.456956 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.456927    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"]
Jul 03 21:04:49.457395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.457373    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8e585c-13e0-4a9d-bb03-69a3ec39e654" containerName="istio-proxy"
Jul 03 21:04:49.457395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.457394    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8e585c-13e0-4a9d-bb03-69a3ec39e654" containerName="istio-proxy"
Jul 03 21:04:49.457574 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.457463    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8e585c-13e0-4a9d-bb03-69a3ec39e654" containerName="istio-proxy"
Jul 03 21:04:49.459962 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.459941    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.463700 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.463680    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ddaab74d-openshift-default-dockercfg-4cn7q\""
Jul 03 21:04:49.478032 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.478009    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"]
Jul 03 21:04:49.502145 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.502096    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:49.522484 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522453    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-credential-socket\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522607 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522492    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-certs\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522607 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522575    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-socket\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522684 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522628    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-podinfo\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522684 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522660    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-data\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522750 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522682    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-token\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522750 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522725    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-envoy\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522750 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522745    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istiod-ca-cert\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.522846 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.522783    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkq9\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-kube-api-access-kfkq9\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.572296 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.572199    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:49.623951 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.623881    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-certs\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624107 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624002    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-socket\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624107 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624047    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-podinfo\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624195 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624166    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-data\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624243 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-token\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624295 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624244    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-envoy\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624295 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624177    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-certs\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624295 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624272    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istiod-ca-cert\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624459 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624314    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkq9\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-kube-api-access-kfkq9\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624459 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624361    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-socket\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624459 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624408    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-credential-socket\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624657 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624531    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-data\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624716 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624697    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-credential-socket\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.624948 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.624928    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istiod-ca-cert\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.626375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.626349    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-envoy\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.626520 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.626499    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-podinfo\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.633219 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.633200    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-token\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.633313 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.633290    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkq9\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-kube-api-access-kfkq9\") pod \"e2e-ait-ddaab74d-openshift-default-6547898677-hx2js\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") " pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.774004 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.773971    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:49.909213 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.908937    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"]
Jul 03 21:04:49.911736 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:04:49.911706    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81bb90e_dfdf_408b_827b_540fcd3f47c3.slice/crio-a07ada4f0fb775f547c6b167fd2f9b7166daded745f13f06280adf3904a3afa0 WatchSource:0}: Error finding container a07ada4f0fb775f547c6b167fd2f9b7166daded745f13f06280adf3904a3afa0: Status 404 returned error can't find the container with id a07ada4f0fb775f547c6b167fd2f9b7166daded745f13f06280adf3904a3afa0
Jul 03 21:04:49.913947 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.913880    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:04:49.914030 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.914000    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:04:49.914069 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.914029    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:04:49.938935 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.938916    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"
Jul 03 21:04:49.939244 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.938915    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" event={"ID":"c81bb90e-dfdf-408b-827b-540fcd3f47c3","Type":"ContainerStarted","Data":"a07ada4f0fb775f547c6b167fd2f9b7166daded745f13f06280adf3904a3afa0"}
Jul 03 21:04:49.967039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.967017    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"]
Jul 03 21:04:49.970196 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:49.970175    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-753c62a5-openshift-default-5cc685d879-d6z9k"]
Jul 03 21:04:50.691308 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.691272    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:50.767024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.766994    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:50.832043 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.832001    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:50.870226 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.870190    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d8e585c-13e0-4a9d-bb03-69a3ec39e654" path="/var/lib/kubelet/pods/2d8e585c-13e0-4a9d-bb03-69a3ec39e654/volumes"
Jul 03 21:04:50.925742 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.925710    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:50.946122 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.946028    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" event={"ID":"c81bb90e-dfdf-408b-827b-540fcd3f47c3","Type":"ContainerStarted","Data":"d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"}
Jul 03 21:04:50.973199 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:50.973136    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" podStartSLOduration=1.9731179760000002 podStartE2EDuration="1.973117976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:04:50.971167272 +0000 UTC m=+1663.051908680" watchObservedRunningTime="2026-07-03 21:04:50.973117976 +0000 UTC m=+1663.053859388"
Jul 03 21:04:51.206748 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:51.206665    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:51.304460 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:51.304429    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:51.775002 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:51.774970    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:51.779617 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:51.779596    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:51.951462 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:51.951432    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:51.952393 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:51.952374    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:04:52.904544 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:52.904511    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:55.789281 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:55.789249    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:04:56.050007 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:04:56.049922    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:02.000142 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.000088    2553 generic.go:356] "Generic (PLEG): container finished" podID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerID="d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913" exitCode=0
Jul 03 21:05:02.000544 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.000179    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" event={"ID":"c81bb90e-dfdf-408b-827b-540fcd3f47c3","Type":"ContainerDied","Data":"d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"}
Jul 03 21:05:02.000697 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.000681    2553 scope.go:117] "RemoveContainer" containerID="d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"
Jul 03 21:05:02.001778 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.001749    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:05:02.001853 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.001836    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:05:02.001929 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.001910    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:05:02.097429 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.097335    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"]
Jul 03 21:05:02.120264 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.120232    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:02.219839 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:02.219808    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:03.007174 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:03.007140    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" event={"ID":"c81bb90e-dfdf-408b-827b-540fcd3f47c3","Type":"ContainerStarted","Data":"bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201"}
Jul 03 21:05:03.007637 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:03.007224    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy" containerID="cri-o://bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201" gracePeriod=30
Jul 03 21:05:04.116733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:04.116695    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:06.952230 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:06.952194    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:05:07.795229 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:07.795197    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:07.895922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:07.895856    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:08.001685 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.001652    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:08.100067 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.099985    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:08.259497 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.259471    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:05:08.399625 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399537    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-data\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399625 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399581    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-credential-socket\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399625 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399614    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-socket\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399642    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istiod-ca-cert\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399682    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfkq9\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-kube-api-access-kfkq9\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399764    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-certs\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399793    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-podinfo\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399816    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-token\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.399923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399838    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-envoy\") pod \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\" (UID: \"c81bb90e-dfdf-408b-827b-540fcd3f47c3\") "
Jul 03 21:05:08.400228 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399923    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:05:08.400228 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.399997    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:05:08.400228 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.400139    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:05:08.400439 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.400419    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 21:05:08.400609 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.400579    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-credential-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.400686 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.400605    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.400686 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.400623    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-workload-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.400686 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.400630    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:05:08.402707 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.402669    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:05:08.402802 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.402725    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:05:08.403038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.403016    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 03 21:05:08.403089 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.403057    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-kube-api-access-kfkq9" (OuterVolumeSpecName: "kube-api-access-kfkq9") pod "c81bb90e-dfdf-408b-827b-540fcd3f47c3" (UID: "c81bb90e-dfdf-408b-827b-540fcd3f47c3"). InnerVolumeSpecName "kube-api-access-kfkq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:05:08.501832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.501793    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkq9\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-kube-api-access-kfkq9\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.501832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.501835    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-podinfo\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.502056 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.501849    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-token\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.502056 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.501866    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-envoy\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.502056 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.501878    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istio-data\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:08.502056 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:08.501911    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81bb90e-dfdf-408b-827b-540fcd3f47c3-istiod-ca-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:05:09.038176 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.038143    2553 generic.go:356] "Generic (PLEG): container finished" podID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerID="bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201" exitCode=0
Jul 03 21:05:09.038624 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.038221    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" event={"ID":"c81bb90e-dfdf-408b-827b-540fcd3f47c3","Type":"ContainerDied","Data":"bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201"}
Jul 03 21:05:09.038624 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.038280    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"
Jul 03 21:05:09.038624 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.038292    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js" event={"ID":"c81bb90e-dfdf-408b-827b-540fcd3f47c3","Type":"ContainerDied","Data":"a07ada4f0fb775f547c6b167fd2f9b7166daded745f13f06280adf3904a3afa0"}
Jul 03 21:05:09.038624 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.038315    2553 scope.go:117] "RemoveContainer" containerID="bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201"
Jul 03 21:05:09.046093 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.046076    2553 scope.go:117] "RemoveContainer" containerID="d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"
Jul 03 21:05:09.053225 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.053206    2553 scope.go:117] "RemoveContainer" containerID="bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201"
Jul 03 21:05:09.053486 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:05:09.053468    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201\": container with ID starting with bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201 not found: ID does not exist" containerID="bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201"
Jul 03 21:05:09.053563 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.053497    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201"} err="failed to get container status \"bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201\": rpc error: code = NotFound desc = could not find container \"bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201\": container with ID starting with bd0a47bc4370ccb733af55fd39f099f34f13c23d090a9155bca42f9b0a9b0201 not found: ID does not exist"
Jul 03 21:05:09.053563 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.053519    2553 scope.go:117] "RemoveContainer" containerID="d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"
Jul 03 21:05:09.053787 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:05:09.053769    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913\": container with ID starting with d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913 not found: ID does not exist" containerID="d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"
Jul 03 21:05:09.053839 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.053793    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913"} err="failed to get container status \"d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913\": rpc error: code = NotFound desc = could not find container \"d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913\": container with ID starting with d2dc3a163de6c70a83fd9170020e9a9cdac08ed15a7c3122527bc60ccf468913 not found: ID does not exist"
Jul 03 21:05:09.058537 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.058516    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"]
Jul 03 21:05:09.064982 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.064963    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ddaab74d-openshift-default-6547898677-hx2js"]
Jul 03 21:05:09.201846 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.201767    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:09.302209 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.302175    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:09.473402 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.473290    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:09.720685 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.720653    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:09.984770 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:09.984741    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:10.086403 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:10.086371    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:10.870915 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:10.870868    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" path="/var/lib/kubelet/pods/c81bb90e-dfdf-408b-827b-540fcd3f47c3/volumes"
Jul 03 21:05:11.334963 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:11.334927    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:13.643013 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:13.642981    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:13.827384 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:13.827354    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:13.941614 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:13.941533    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:14.049915 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:14.049860    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:15.335574 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:15.335543    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:19.462366 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:19.462317    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:19.563333 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:19.563287    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:19.931492 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:19.931440    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:20.030063 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:20.030032    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:20.154394 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:20.154357    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:21.505917 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:21.504447    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:27.272654 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:27.272620    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:27.329511 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:27.329472    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:27.406341 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:27.406290    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:27.502230 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:27.502197    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:27.602862 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:27.602817    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:28.015610 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:28.015519    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:30.666519 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:30.666442    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:30.766648 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:30.766618    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:53.905575 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:53.905541    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:59.646099 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:59.646055    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:59.747444 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:59.747412    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:05:59.845664 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:05:59.845623    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:15.740528 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:15.740482    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:21.507066 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:21.507033    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:21.567057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:21.567022    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:21.641707 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:21.641668    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:21.744797 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:21.744761    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:32.241958 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:32.241920    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:33.014509 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:33.014479    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:38.752692 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:38.752654    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:38.857300 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:38.857267    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:38.953134 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:38.953101    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:39.053035 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:39.053003    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:39.809236 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:39.809196    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:39.906278 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:39.906247    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:40.007193 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:40.007156    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:41.567507 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:41.567477    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:47.411205 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:47.411163    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:47.473404 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:47.473369    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:48.069151 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:48.069115    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:53.808069 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:53.808032    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:53.908105 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:53.908060    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:54.035653 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:54.035625    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:54.359750 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:54.359722    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:55.038319 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:55.038275    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:57.773422 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:57.773388    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:57.864604 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:57.864575    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:57.965699 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:57.965661    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:58.068690 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:58.068611    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:58.315011 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:58.314982    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:06:58.417069 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:06:58.416996    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:02.266981 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:02.266886    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:02.912083 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:02.912044    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:03.011559 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:03.011522    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:03.342030 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:03.341993    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:09.022402 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:09.022375    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:09.121838 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:09.121810    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:09.416732 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:09.416688    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:09.701334 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:09.701218    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:09.832962 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:09.832877    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:15.644346 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:15.644311    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:15.748719 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:15.748688    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:15.919924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:15.919816    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:16.168175 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:16.168143    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:16.286413 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:16.286379    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:16.385066 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:16.385027    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:16.490274 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:16.490230    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:22.480017 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:22.479965    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:22.546342 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:22.546312    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:22.635356 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:22.635322    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:22.735761 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:22.735677    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:22.838213 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:22.838173    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:28.493119 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:28.493078    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:28.592916 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:28.592866    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:34.873474 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:34.873434    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:35.169952 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:35.169840    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:35.398071 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:35.398024    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:38.054846 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:38.054805    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:38.153521 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:38.153488    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:38.262543 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:38.262512    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:38.457219 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:38.457119    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:38.703070 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:38.703036    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:38.798954 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:38.798917    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:42.323547 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:42.323514    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:42.557133 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:42.557106    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:47.349906 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.349866    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"]
Jul 03 21:07:47.350352 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.350334    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy"
Jul 03 21:07:47.350352 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.350354    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy"
Jul 03 21:07:47.350491 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.350363    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy"
Jul 03 21:07:47.350491 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.350369    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy"
Jul 03 21:07:47.350491 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.350448    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy"
Jul 03 21:07:47.350491 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.350457    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81bb90e-dfdf-408b-827b-540fcd3f47c3" containerName="istio-proxy"
Jul 03 21:07:47.353446 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.353426    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.357663 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.357638    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73d964af\"/\"e2e-lifecycle-model-73d964af-kserve-self-signed-certs\""
Jul 03 21:07:47.357663 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.357650    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73d964af\"/\"kube-root-ca.crt\""
Jul 03 21:07:47.357845 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.357711    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73d964af\"/\"default-dockercfg-9vv5x\""
Jul 03 21:07:47.357845 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.357742    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73d964af\"/\"openshift-service-ca.crt\""
Jul 03 21:07:47.363434 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.363407    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"]
Jul 03 21:07:47.378872 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.378836    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:07:47.469572 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.469541    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-home\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.469734 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.469579    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24127bf2-7860-4604-a5e6-11ab5537dba3-tls-certs\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.469734 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.469599    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-tmp-dir\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.469734 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.469716    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-model-cache\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.469840 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.469757    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4hq\" (UniqueName: \"kubernetes.io/projected/24127bf2-7860-4604-a5e6-11ab5537dba3-kube-api-access-ln4hq\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.469840 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.469777    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-dshm\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.570863 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.570832    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-home\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.570874    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24127bf2-7860-4604-a5e6-11ab5537dba3-tls-certs\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.570940    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-tmp-dir\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.571008    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-model-cache\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.571041    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4hq\" (UniqueName: \"kubernetes.io/projected/24127bf2-7860-4604-a5e6-11ab5537dba3-kube-api-access-ln4hq\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571290 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.571072    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-dshm\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571346 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.571284    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-home\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571401 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.571342    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-tmp-dir\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.571401 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.571381    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-model-cache\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.573247 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.573226    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-dshm\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.573617 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.573598    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24127bf2-7860-4604-a5e6-11ab5537dba3-tls-certs\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.579271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.579247    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4hq\" (UniqueName: \"kubernetes.io/projected/24127bf2-7860-4604-a5e6-11ab5537dba3-kube-api-access-ln4hq\") pod \"e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") " pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.663817 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.663736    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:47.798068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:47.797978    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"]
Jul 03 21:07:47.801326 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:07:47.801291    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24127bf2_7860_4604_a5e6_11ab5537dba3.slice/crio-60a9a0f37e40f921e0f322dbfc21d93b1e74e1e17f467d6f4a69c219284bd7c9 WatchSource:0}: Error finding container 60a9a0f37e40f921e0f322dbfc21d93b1e74e1e17f467d6f4a69c219284bd7c9: Status 404 returned error can't find the container with id 60a9a0f37e40f921e0f322dbfc21d93b1e74e1e17f467d6f4a69c219284bd7c9
Jul 03 21:07:48.779354 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:48.779313    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4" event={"ID":"24127bf2-7860-4604-a5e6-11ab5537dba3","Type":"ContainerStarted","Data":"0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48"}
Jul 03 21:07:48.779739 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:48.779369    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4" event={"ID":"24127bf2-7860-4604-a5e6-11ab5537dba3","Type":"ContainerStarted","Data":"60a9a0f37e40f921e0f322dbfc21d93b1e74e1e17f467d6f4a69c219284bd7c9"}
Jul 03 21:07:48.796251 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:48.796193    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4" podStartSLOduration=1.796177368 podStartE2EDuration="1.796177368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:07:48.795179669 +0000 UTC m=+1840.875921077" watchObservedRunningTime="2026-07-03 21:07:48.796177368 +0000 UTC m=+1840.876918776"
Jul 03 21:07:57.664243 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:57.664204    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:57.664657 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:57.664263    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:57.676681 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:57.676658    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:07:57.833552 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:07:57.833519    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:08:02.082602 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:02.082563    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:02.180802 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:02.180766    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:04.593543 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:04.593430    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:05.069638 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:05.069608    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:05.171459 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:05.171426    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:05.290886 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:05.290849    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:10.668155 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:10.668109    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:15.649525 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:15.649489    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"]
Jul 03 21:08:15.691548 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:15.691513    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:16.864208 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:16.864170    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:30.670354 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.670245    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4" podUID="24127bf2-7860-4604-a5e6-11ab5537dba3" containerName="main" containerID="cri-o://0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48" gracePeriod=45
Jul 03 21:08:30.923950 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.923880    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:08:30.981468 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.981425    2553 generic.go:356] "Generic (PLEG): container finished" podID="24127bf2-7860-4604-a5e6-11ab5537dba3" containerID="0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48" exitCode=0
Jul 03 21:08:30.981468 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.981468    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"
Jul 03 21:08:30.981668 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.981502    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4" event={"ID":"24127bf2-7860-4604-a5e6-11ab5537dba3","Type":"ContainerDied","Data":"0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48"}
Jul 03 21:08:30.981668 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.981555    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4" event={"ID":"24127bf2-7860-4604-a5e6-11ab5537dba3","Type":"ContainerDied","Data":"60a9a0f37e40f921e0f322dbfc21d93b1e74e1e17f467d6f4a69c219284bd7c9"}
Jul 03 21:08:30.981668 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.981573    2553 scope.go:117] "RemoveContainer" containerID="0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48"
Jul 03 21:08:30.990622 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.990602    2553 scope.go:117] "RemoveContainer" containerID="0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48"
Jul 03 21:08:30.990863 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:08:30.990845    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48\": container with ID starting with 0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48 not found: ID does not exist" containerID="0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48"
Jul 03 21:08:30.990947 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:30.990875    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48"} err="failed to get container status \"0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48\": rpc error: code = NotFound desc = could not find container \"0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48\": container with ID starting with 0c2ef26a7abc432a2baaa09e50435ea8b4572cf0f382ceaa52705a3ee7375f48 not found: ID does not exist"
Jul 03 21:08:31.056255 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056229    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24127bf2-7860-4604-a5e6-11ab5537dba3-tls-certs\") pod \"24127bf2-7860-4604-a5e6-11ab5537dba3\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") "
Jul 03 21:08:31.056435 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056272    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-dshm\") pod \"24127bf2-7860-4604-a5e6-11ab5537dba3\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") "
Jul 03 21:08:31.056435 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056320    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-tmp-dir\") pod \"24127bf2-7860-4604-a5e6-11ab5537dba3\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") "
Jul 03 21:08:31.056435 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056342    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln4hq\" (UniqueName: \"kubernetes.io/projected/24127bf2-7860-4604-a5e6-11ab5537dba3-kube-api-access-ln4hq\") pod \"24127bf2-7860-4604-a5e6-11ab5537dba3\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") "
Jul 03 21:08:31.056435 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056372    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-model-cache\") pod \"24127bf2-7860-4604-a5e6-11ab5537dba3\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") "
Jul 03 21:08:31.056435 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056428    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-home\") pod \"24127bf2-7860-4604-a5e6-11ab5537dba3\" (UID: \"24127bf2-7860-4604-a5e6-11ab5537dba3\") "
Jul 03 21:08:31.056694 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056631    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24127bf2-7860-4604-a5e6-11ab5537dba3" (UID: "24127bf2-7860-4604-a5e6-11ab5537dba3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:08:31.056694 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056682    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-model-cache" (OuterVolumeSpecName: "model-cache") pod "24127bf2-7860-4604-a5e6-11ab5537dba3" (UID: "24127bf2-7860-4604-a5e6-11ab5537dba3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:08:31.056832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.056811    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-home" (OuterVolumeSpecName: "home") pod "24127bf2-7860-4604-a5e6-11ab5537dba3" (UID: "24127bf2-7860-4604-a5e6-11ab5537dba3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:08:31.058571 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.058545    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-dshm" (OuterVolumeSpecName: "dshm") pod "24127bf2-7860-4604-a5e6-11ab5537dba3" (UID: "24127bf2-7860-4604-a5e6-11ab5537dba3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:08:31.058667 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.058611    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24127bf2-7860-4604-a5e6-11ab5537dba3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24127bf2-7860-4604-a5e6-11ab5537dba3" (UID: "24127bf2-7860-4604-a5e6-11ab5537dba3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:08:31.058707 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.058678    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24127bf2-7860-4604-a5e6-11ab5537dba3-kube-api-access-ln4hq" (OuterVolumeSpecName: "kube-api-access-ln4hq") pod "24127bf2-7860-4604-a5e6-11ab5537dba3" (UID: "24127bf2-7860-4604-a5e6-11ab5537dba3"). InnerVolumeSpecName "kube-api-access-ln4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:08:31.157908 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.157875    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-tmp-dir\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:08:31.157908 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.157912    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4hq\" (UniqueName: \"kubernetes.io/projected/24127bf2-7860-4604-a5e6-11ab5537dba3-kube-api-access-ln4hq\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:08:31.158087 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.157922    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-model-cache\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:08:31.158087 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.157930    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-home\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:08:31.158087 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.157937    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24127bf2-7860-4604-a5e6-11ab5537dba3-tls-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:08:31.158087 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.157944    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24127bf2-7860-4604-a5e6-11ab5537dba3-dshm\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:08:31.307159 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.304546    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"]
Jul 03 21:08:31.312719 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:31.312690    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-73d964af/e2e-lifecycle-model-73d964af-kserve-8c867fb75-b6mw4"]
Jul 03 21:08:32.870272 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:32.870242    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24127bf2-7860-4604-a5e6-11ab5537dba3" path="/var/lib/kubelet/pods/24127bf2-7860-4604-a5e6-11ab5537dba3/volumes"
Jul 03 21:08:51.835132 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:51.835092    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:53.000184 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:53.000147    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:58.434583 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:58.434546    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:58.534800 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:58.534771    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:58.703166 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:58.703088    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:58.932594 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:58.932556    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:08:59.966832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:08:59.966775    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:00.064845 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:00.064810    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:00.164501 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:00.164471    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:00.863679 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:00.863638    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:00.962658 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:00.962625    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:01.062339 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:01.062306    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:02.426058 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:02.426014    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:08.404387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:08.404349    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:09.492860 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:09.492821    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:14.772602 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:14.772555    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:14.832164 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:14.832127    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:14.905260 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:14.905229    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:15.006558 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:15.006525    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:25.492008 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:25.491977    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:25.845635 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:25.845601    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:46.792236 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:46.792199    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:52.333680 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:52.333642    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:52.434461 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:52.434415    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:52.692970 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:52.692851    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:52.993318 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:52.993240    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:53.230068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:53.230037    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:55.783001 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:55.782970    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:55.871221 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:55.871185    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:55.937410 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:55.937377    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:56.094098 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:56.093989    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:56.330736 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:56.330688    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:09:56.432370 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:09:56.432237    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:00.318174 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.318094    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:00.666208 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.666105    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:00.942302 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.942206    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"]
Jul 03 21:10:00.942693 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.942667    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24127bf2-7860-4604-a5e6-11ab5537dba3" containerName="main"
Jul 03 21:10:00.942693 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.942690    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="24127bf2-7860-4604-a5e6-11ab5537dba3" containerName="main"
Jul 03 21:10:00.942910 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.942762    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="24127bf2-7860-4604-a5e6-11ab5537dba3" containerName="main"
Jul 03 21:10:00.945758 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.945740    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:00.948553 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.948535    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-33b41b-openshift-default-dockercfg-755tr\""
Jul 03 21:10:00.956611 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.956590    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"]
Jul 03 21:10:00.973938 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:00.973916    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:01.017672 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017637    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zwx\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-kube-api-access-t2zwx\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.017829 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017678    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-socket\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.017829 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017702    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-data\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.017829 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017749    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-podinfo\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.017829 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017805    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-certs\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.018057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017842    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-envoy\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.018057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017858    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istiod-ca-cert\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.018057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.017971    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-token\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.018057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.018015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-credential-socket\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-credential-socket\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118401    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zwx\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-kube-api-access-t2zwx\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118615 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118429    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-socket\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118615 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118453    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-data\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118615 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118480    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-podinfo\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118615 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118527    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-certs\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118649    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-envoy\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istiod-ca-cert\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.118836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118753    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-token\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.119027 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118880    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-credential-socket\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.119027 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.118959    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-data\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.119155 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.119040    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-socket\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.119247 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.119227    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-certs\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.119477 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.119456    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istiod-ca-cert\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.121093 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.121070    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-envoy\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.121200 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.121070    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-podinfo\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.126400 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.126371    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-token\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.126580 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.126562    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zwx\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-kube-api-access-t2zwx\") pod \"e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") " pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.256852 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.256766    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:01.409819 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.409795    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"]
Jul 03 21:10:01.412365 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:10:01.412335    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc3b979_195e_4a48_b04f_3ea0a2e7598d.slice/crio-fc06e4d8e1a90b8b4d665f9066d0c117b6e00afd6bfcac13d51f52b50279790c WatchSource:0}: Error finding container fc06e4d8e1a90b8b4d665f9066d0c117b6e00afd6bfcac13d51f52b50279790c: Status 404 returned error can't find the container with id fc06e4d8e1a90b8b4d665f9066d0c117b6e00afd6bfcac13d51f52b50279790c
Jul 03 21:10:01.414465 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.414431    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:10:01.414567 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.414509    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:10:01.414567 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.414553    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:10:01.488398 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:01.488368    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:02.408458 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:02.408420    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" event={"ID":"bbc3b979-195e-4a48-b04f-3ea0a2e7598d","Type":"ContainerStarted","Data":"4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"}
Jul 03 21:10:02.408638 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:02.408511    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" event={"ID":"bbc3b979-195e-4a48-b04f-3ea0a2e7598d","Type":"ContainerStarted","Data":"fc06e4d8e1a90b8b4d665f9066d0c117b6e00afd6bfcac13d51f52b50279790c"}
Jul 03 21:10:02.429868 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:02.429816    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podStartSLOduration=2.42980136 podStartE2EDuration="2.42980136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:10:02.427703907 +0000 UTC m=+1974.508445298" watchObservedRunningTime="2026-07-03 21:10:02.42980136 +0000 UTC m=+1974.510542767"
Jul 03 21:10:03.257331 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:03.257278    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:03.262540 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:03.262510    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:03.414106 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:03.414069    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:03.415056 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:03.415036    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:04.483405 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:04.483367    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:04.578270 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:04.578174    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:04.650093 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:04.650056    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:04.750224 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:04.750179    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:05.158163 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:05.158127    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:05.252134 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:05.252093    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:08.897108 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:08.897067    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:17.081736 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:17.081694    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:17.181128 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:17.181087    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:18.415052 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.415019    2553 patch_prober.go:28] interesting pod/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 03 21:10:18.415457 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.415070    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 03 21:10:18.483794 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.483759    2553 generic.go:356] "Generic (PLEG): container finished" podID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerID="4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d" exitCode=0
Jul 03 21:10:18.483987 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.483833    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" event={"ID":"bbc3b979-195e-4a48-b04f-3ea0a2e7598d","Type":"ContainerDied","Data":"4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"}
Jul 03 21:10:18.484261 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.484247    2553 scope.go:117] "RemoveContainer" containerID="4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"
Jul 03 21:10:18.485315 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.485284    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:10:18.485391 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.485371    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:10:18.485432 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:18.485417    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:10:19.491869 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:19.491839    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" event={"ID":"bbc3b979-195e-4a48-b04f-3ea0a2e7598d","Type":"ContainerStarted","Data":"ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220"}
Jul 03 21:10:20.257769 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:20.257730    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:20.258000 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:20.257964    2553 patch_prober.go:28] interesting pod/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 03 21:10:20.258058 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:20.258010    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 03 21:10:21.257651 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:21.257608    2553 patch_prober.go:28] interesting pod/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 03 21:10:21.258088 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:21.257668    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 03 21:10:22.257830 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:22.257796    2553 patch_prober.go:28] interesting pod/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 03 21:10:22.258237 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:22.257866    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 03 21:10:23.257969 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:23.257928    2553 patch_prober.go:28] interesting pod/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Jul 03 21:10:23.258396 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:23.257984    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Jul 03 21:10:24.261176 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:24.261145    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:24.261543 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:24.261429    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:24.262156 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:24.262137    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:10:25.801222 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:25.801161    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:25.902427 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:25.902386    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:29.244022 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.243991    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"]
Jul 03 21:10:29.248222 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.248202    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.252423 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.252395    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-12615b\"/\"kube-root-ca.crt\""
Jul 03 21:10:29.252554 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.252434    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-12615b\"/\"test-model-12615b-kserve-self-signed-certs\""
Jul 03 21:10:29.252554 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.252400    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-12615b\"/\"default-dockercfg-krdzp\""
Jul 03 21:10:29.252720 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.252412    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-12615b\"/\"openshift-service-ca.crt\""
Jul 03 21:10:29.257956 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.257925    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"]
Jul 03 21:10:29.278080 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.278049    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:29.363097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.363067    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-tmp-dir\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.363271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.363111    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-home\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.363271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.363136    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc13085-0f45-432f-9e89-eef1d2467b86-tls-certs\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.363271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.363174    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9p6j\" (UniqueName: \"kubernetes.io/projected/6bc13085-0f45-432f-9e89-eef1d2467b86-kube-api-access-z9p6j\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.363271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.363207    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-dshm\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.363271 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.363250    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-model-cache\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.464671 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.464631    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-dshm\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.464871 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.464681    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-model-cache\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.464871 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.464737    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-tmp-dir\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.464871 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.464790    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-home\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.464871 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.464827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc13085-0f45-432f-9e89-eef1d2467b86-tls-certs\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.464871 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.464860    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9p6j\" (UniqueName: \"kubernetes.io/projected/6bc13085-0f45-432f-9e89-eef1d2467b86-kube-api-access-z9p6j\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.465176 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.465127    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-model-cache\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.465245 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.465196    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-tmp-dir\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.465245 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.465212    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-home\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.467104 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.467079    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-dshm\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.467291 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.467273    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc13085-0f45-432f-9e89-eef1d2467b86-tls-certs\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.473984 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.473948    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9p6j\" (UniqueName: \"kubernetes.io/projected/6bc13085-0f45-432f-9e89-eef1d2467b86-kube-api-access-z9p6j\") pod \"test-model-12615b-kserve-5c968cf8b7-mjvpw\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") " pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.558905 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.558865    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:29.700415 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:29.700386    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"]
Jul 03 21:10:29.702861 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:10:29.702832    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bc13085_0f45_432f_9e89_eef1d2467b86.slice/crio-8ef92ec71245bdaf615b7aaeef68e0b446da9ea900c5ea3869dec2d590cfcff7 WatchSource:0}: Error finding container 8ef92ec71245bdaf615b7aaeef68e0b446da9ea900c5ea3869dec2d590cfcff7: Status 404 returned error can't find the container with id 8ef92ec71245bdaf615b7aaeef68e0b446da9ea900c5ea3869dec2d590cfcff7
Jul 03 21:10:30.553149 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:30.553117    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw" event={"ID":"6bc13085-0f45-432f-9e89-eef1d2467b86","Type":"ContainerStarted","Data":"e5ac9b1d54665808daa91300921d8b294673c66f5870912487c782dbbba43452"}
Jul 03 21:10:30.553149 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:30.553157    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw" event={"ID":"6bc13085-0f45-432f-9e89-eef1d2467b86","Type":"ContainerStarted","Data":"8ef92ec71245bdaf615b7aaeef68e0b446da9ea900c5ea3869dec2d590cfcff7"}
Jul 03 21:10:30.571867 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:30.571813    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw" podStartSLOduration=1.5717988109999999 podStartE2EDuration="1.571798811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:10:30.570937565 +0000 UTC m=+2002.651678974" watchObservedRunningTime="2026-07-03 21:10:30.571798811 +0000 UTC m=+2002.652540219"
Jul 03 21:10:39.559297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:39.559252    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:39.559822 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:39.559333    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:39.572163 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:39.572135    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:39.611553 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:39.611525    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:10:55.525124 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:55.525089    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:56.049585 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:56.049553    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:10:56.103705 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:10:56.103667    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:08.564561 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:08.564529    2553 scope.go:117] "RemoveContainer" containerID="54c0628b460bb36f7eb03ab970ea22a26727a02a6172d50c9265d767e32dd66a"
Jul 03 21:11:17.688476 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:17.688445    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"]
Jul 03 21:11:17.706482 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:17.706454    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:18.652200 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:18.652158    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:18.804671 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:18.804635    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:19.185337 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:19.185303    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:19.285153 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:19.285118    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:19.499791 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:19.499712    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:24.296247 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:24.296216    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:32.714541 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:32.714448    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw" podUID="6bc13085-0f45-432f-9e89-eef1d2467b86" containerName="main" containerID="cri-o://e5ac9b1d54665808daa91300921d8b294673c66f5870912487c782dbbba43452" gracePeriod=45
Jul 03 21:11:32.857394 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:32.857358    2553 generic.go:356] "Generic (PLEG): container finished" podID="6bc13085-0f45-432f-9e89-eef1d2467b86" containerID="e5ac9b1d54665808daa91300921d8b294673c66f5870912487c782dbbba43452" exitCode=0
Jul 03 21:11:32.857580 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:32.857413    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw" event={"ID":"6bc13085-0f45-432f-9e89-eef1d2467b86","Type":"ContainerDied","Data":"e5ac9b1d54665808daa91300921d8b294673c66f5870912487c782dbbba43452"}
Jul 03 21:11:32.978511 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:32.978454    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:11:33.135917 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.135850    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9p6j\" (UniqueName: \"kubernetes.io/projected/6bc13085-0f45-432f-9e89-eef1d2467b86-kube-api-access-z9p6j\") pod \"6bc13085-0f45-432f-9e89-eef1d2467b86\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") "
Jul 03 21:11:33.136097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.135946    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-dshm\") pod \"6bc13085-0f45-432f-9e89-eef1d2467b86\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") "
Jul 03 21:11:33.136097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.135976    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-model-cache\") pod \"6bc13085-0f45-432f-9e89-eef1d2467b86\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") "
Jul 03 21:11:33.136097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.136014    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-home\") pod \"6bc13085-0f45-432f-9e89-eef1d2467b86\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") "
Jul 03 21:11:33.136097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.136042    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-tmp-dir\") pod \"6bc13085-0f45-432f-9e89-eef1d2467b86\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") "
Jul 03 21:11:33.136097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.136066    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc13085-0f45-432f-9e89-eef1d2467b86-tls-certs\") pod \"6bc13085-0f45-432f-9e89-eef1d2467b86\" (UID: \"6bc13085-0f45-432f-9e89-eef1d2467b86\") "
Jul 03 21:11:33.136364 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.136276    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-home" (OuterVolumeSpecName: "home") pod "6bc13085-0f45-432f-9e89-eef1d2467b86" (UID: "6bc13085-0f45-432f-9e89-eef1d2467b86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:11:33.136364 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.136288    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bc13085-0f45-432f-9e89-eef1d2467b86" (UID: "6bc13085-0f45-432f-9e89-eef1d2467b86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:11:33.136450 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.136368    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bc13085-0f45-432f-9e89-eef1d2467b86" (UID: "6bc13085-0f45-432f-9e89-eef1d2467b86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:11:33.138161 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.138140    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-dshm" (OuterVolumeSpecName: "dshm") pod "6bc13085-0f45-432f-9e89-eef1d2467b86" (UID: "6bc13085-0f45-432f-9e89-eef1d2467b86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:11:33.138258 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.138235    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc13085-0f45-432f-9e89-eef1d2467b86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bc13085-0f45-432f-9e89-eef1d2467b86" (UID: "6bc13085-0f45-432f-9e89-eef1d2467b86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:11:33.138297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.138266    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc13085-0f45-432f-9e89-eef1d2467b86-kube-api-access-z9p6j" (OuterVolumeSpecName: "kube-api-access-z9p6j") pod "6bc13085-0f45-432f-9e89-eef1d2467b86" (UID: "6bc13085-0f45-432f-9e89-eef1d2467b86"). InnerVolumeSpecName "kube-api-access-z9p6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:11:33.237297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.237208    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9p6j\" (UniqueName: \"kubernetes.io/projected/6bc13085-0f45-432f-9e89-eef1d2467b86-kube-api-access-z9p6j\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:11:33.237297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.237241    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-dshm\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:11:33.237297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.237253    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-model-cache\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:11:33.237297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.237265    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-home\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:11:33.237297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.237275    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc13085-0f45-432f-9e89-eef1d2467b86-tmp-dir\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:11:33.237297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.237285    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc13085-0f45-432f-9e89-eef1d2467b86-tls-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:11:33.863008 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.862981    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"
Jul 03 21:11:33.863432 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.863005    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw" event={"ID":"6bc13085-0f45-432f-9e89-eef1d2467b86","Type":"ContainerDied","Data":"8ef92ec71245bdaf615b7aaeef68e0b446da9ea900c5ea3869dec2d590cfcff7"}
Jul 03 21:11:33.863432 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.863051    2553 scope.go:117] "RemoveContainer" containerID="e5ac9b1d54665808daa91300921d8b294673c66f5870912487c782dbbba43452"
Jul 03 21:11:33.884501 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.884477    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"]
Jul 03 21:11:33.886090 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:33.886072    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-12615b/test-model-12615b-kserve-5c968cf8b7-mjvpw"]
Jul 03 21:11:34.870749 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:34.870711    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc13085-0f45-432f-9e89-eef1d2467b86" path="/var/lib/kubelet/pods/6bc13085-0f45-432f-9e89-eef1d2467b86/volumes"
Jul 03 21:11:53.338220 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:53.338178    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:53.612458 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:53.612365    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:53.824516 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:53.824477    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:53.989428 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:53.989329    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:54.364452 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:54.364424    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:54.465459 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:54.465430    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:54.581499 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:54.581459    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:11:59.977491 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:11:59.977459    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:00.050015 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:00.049979    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:00.221755 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:00.221726    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"]
Jul 03 21:12:00.221958 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:00.221935    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy" containerID="cri-o://ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220" gracePeriod=30
Jul 03 21:12:00.247348 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:00.247279    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:00.458350 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:00.458320    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:00.799875 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:00.799842    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:01.097979 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:01.097873    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:01.278528 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:01.278496    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:04.659352 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:04.659199    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:04.732329 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:04.732297    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:04.792290 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:04.792212    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:04.892213 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:04.892177    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:05.292919 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.292868    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:05.393467 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.393437    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:05.483439 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.483373    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:12:05.523718 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523687    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-token\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.523875 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523733    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-podinfo\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.523875 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523758    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-envoy\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.523875 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523775    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-credential-socket\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.523875 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523833    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-socket\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.523875 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523852    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-certs\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.524173 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523915    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2zwx\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-kube-api-access-t2zwx\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.524173 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.523940    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istiod-ca-cert\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.524173 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524026    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-data\") pod \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\" (UID: \"bbc3b979-195e-4a48-b04f-3ea0a2e7598d\") "
Jul 03 21:12:05.524173 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524142    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:05.524386 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524175    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:05.524386 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524359    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-credential-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.524386 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524379    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.524557 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524532    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:05.524724 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524698    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 21:12:05.524958 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.524934    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:05.526164 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.526139    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:12:05.526164 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.526148    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:05.526360 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.526337    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-kube-api-access-t2zwx" (OuterVolumeSpecName: "kube-api-access-t2zwx") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "kube-api-access-t2zwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:12:05.526452 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.526432    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbc3b979-195e-4a48-b04f-3ea0a2e7598d" (UID: "bbc3b979-195e-4a48-b04f-3ea0a2e7598d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 03 21:12:05.625825 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625783    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-podinfo\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.625825 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625819    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-envoy\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.625825 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625833    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-workload-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.626114 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625847    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2zwx\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-kube-api-access-t2zwx\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.626114 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625861    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istiod-ca-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.626114 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625873    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-data\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:05.626114 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:05.625883    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc3b979-195e-4a48-b04f-3ea0a2e7598d-istio-token\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:06.015689 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.015654    2553 generic.go:356] "Generic (PLEG): container finished" podID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerID="ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220" exitCode=0
Jul 03 21:12:06.016189 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.015699    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"
Jul 03 21:12:06.016189 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.015707    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" event={"ID":"bbc3b979-195e-4a48-b04f-3ea0a2e7598d","Type":"ContainerDied","Data":"ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220"}
Jul 03 21:12:06.016189 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.015767    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7" event={"ID":"bbc3b979-195e-4a48-b04f-3ea0a2e7598d","Type":"ContainerDied","Data":"fc06e4d8e1a90b8b4d665f9066d0c117b6e00afd6bfcac13d51f52b50279790c"}
Jul 03 21:12:06.016189 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.015801    2553 scope.go:117] "RemoveContainer" containerID="ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220"
Jul 03 21:12:06.025841 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.025824    2553 scope.go:117] "RemoveContainer" containerID="4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"
Jul 03 21:12:06.033439 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.033422    2553 scope.go:117] "RemoveContainer" containerID="ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220"
Jul 03 21:12:06.033685 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:12:06.033667    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220\": container with ID starting with ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220 not found: ID does not exist" containerID="ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220"
Jul 03 21:12:06.033727 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.033693    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220"} err="failed to get container status \"ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220\": rpc error: code = NotFound desc = could not find container \"ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220\": container with ID starting with ba6b3afa78a1a19bf5df94bd1502982d0dba0d9cb6b7c6f35fe18cb0e7dc1220 not found: ID does not exist"
Jul 03 21:12:06.033727 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.033710    2553 scope.go:117] "RemoveContainer" containerID="4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"
Jul 03 21:12:06.033960 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:12:06.033942    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d\": container with ID starting with 4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d not found: ID does not exist" containerID="4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"
Jul 03 21:12:06.034014 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.033966    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d"} err="failed to get container status \"4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d\": rpc error: code = NotFound desc = could not find container \"4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d\": container with ID starting with 4665da3c83bdba57f1c2ad8b714ea0ba4b4e01a4ee7ee9936cdc217e9270a25d not found: ID does not exist"
Jul 03 21:12:06.037817 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.037796    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"]
Jul 03 21:12:06.040924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.040885    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-33b41b-openshift-default-5bcfb5d545-t8sv7"]
Jul 03 21:12:06.871014 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:06.870983    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" path="/var/lib/kubelet/pods/bbc3b979-195e-4a48-b04f-3ea0a2e7598d/volumes"
Jul 03 21:12:09.053106 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.053078    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:09.522600 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.522516    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:09.781611 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.781583    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"]
Jul 03 21:12:09.781976 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.781957    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy"
Jul 03 21:12:09.781976 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.781973    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy"
Jul 03 21:12:09.782186 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.782000    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc13085-0f45-432f-9e89-eef1d2467b86" containerName="main"
Jul 03 21:12:09.782186 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.782006    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc13085-0f45-432f-9e89-eef1d2467b86" containerName="main"
Jul 03 21:12:09.782186 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.782063    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy"
Jul 03 21:12:09.782186 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.782073    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc13085-0f45-432f-9e89-eef1d2467b86" containerName="main"
Jul 03 21:12:09.782186 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.782084    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy"
Jul 03 21:12:09.786819 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.786799    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.789403 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.789384    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-910755-openshift-default-dockercfg-dgls8\""
Jul 03 21:12:09.797325 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.797299    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"]
Jul 03 21:12:09.818424 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.818390    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:09.864108 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864076    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-podinfo\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864258 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864113    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-certs\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864258 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864137    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-token\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864253    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-socket\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864294    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-data\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864354    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istiod-ca-cert\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864534 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864417    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-credential-socket\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864534 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864469    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-envoy\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.864534 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.864498    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jd5p\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-kube-api-access-9jd5p\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.955566 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.955531    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:09.965716 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965693    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jd5p\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-kube-api-access-9jd5p\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.965840 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965734    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-podinfo\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.965840 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965756    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-certs\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.965840 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965777    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-token\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.965840 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965820    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-socket\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966089 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965846    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-data\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966089 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965917    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istiod-ca-cert\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966089 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.965975    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-credential-socket\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966089 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.966014    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-envoy\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966303 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.966279    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-certs\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966355 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.966310    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-socket\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966416 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.966351    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-data\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966475 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.966437    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-credential-socket\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.966644 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.966623    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istiod-ca-cert\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.968290 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.968269    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-envoy\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.968394 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.968357    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-podinfo\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.973434 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.973404    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-token\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:09.973692 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:09.973668    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jd5p\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-kube-api-access-9jd5p\") pod \"e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") " pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:10.099100 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:10.099000    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:10.242281 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:10.242199    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"]
Jul 03 21:12:10.251875 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:12:10.251843    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85637cc_b4dd_47ea_bd7b_cd3395b524b5.slice/crio-af5d798c3ba97b2792732614fe0af90f83b31db9451a03685ae01a85764fca54 WatchSource:0}: Error finding container af5d798c3ba97b2792732614fe0af90f83b31db9451a03685ae01a85764fca54: Status 404 returned error can't find the container with id af5d798c3ba97b2792732614fe0af90f83b31db9451a03685ae01a85764fca54
Jul 03 21:12:10.254084 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:10.254048    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:10.254175 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:10.254116    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:10.254175 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:10.254145    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:11.041180 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:11.041148    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" event={"ID":"e85637cc-b4dd-47ea-bd7b-cd3395b524b5","Type":"ContainerStarted","Data":"19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"}
Jul 03 21:12:11.041371 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:11.041207    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" event={"ID":"e85637cc-b4dd-47ea-bd7b-cd3395b524b5","Type":"ContainerStarted","Data":"af5d798c3ba97b2792732614fe0af90f83b31db9451a03685ae01a85764fca54"}
Jul 03 21:12:11.061732 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:11.061687    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" podStartSLOduration=2.061672978 podStartE2EDuration="2.061672978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:12:11.059884748 +0000 UTC m=+2103.140626155" watchObservedRunningTime="2026-07-03 21:12:11.061672978 +0000 UTC m=+2103.142414385"
Jul 03 21:12:11.100166 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:11.100133    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:11.105789 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:11.105766    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:12.046231 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:12.046200    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:12.047222 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:12.047201    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:13.058415 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:13.058378    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:13.141522 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:13.141488    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:13.197966 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:13.197935    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:13.370180 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:13.370095    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:13.700575 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:13.700477    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:13.799850 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:13.799791    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:17.550147 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:17.550109    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:25.620582 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:25.620551    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:25.928143 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:25.928057    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:26.024381 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:26.024344    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:26.079215 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:26.079179    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:26.153410 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:26.153359    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:27.047436 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.047404    2553 patch_prober.go:28] interesting pod/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 03 21:12:27.047830 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.047470    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 03 21:12:27.130789 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.130753    2553 generic.go:356] "Generic (PLEG): container finished" podID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerID="19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4" exitCode=0
Jul 03 21:12:27.130979 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.130825    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" event={"ID":"e85637cc-b4dd-47ea-bd7b-cd3395b524b5","Type":"ContainerDied","Data":"19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"}
Jul 03 21:12:27.131220 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.131207    2553 scope.go:117] "RemoveContainer" containerID="19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"
Jul 03 21:12:27.132190 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.132161    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:27.132293 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.132244    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:27.132358 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:27.132294    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:28.137190 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:28.137141    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" event={"ID":"e85637cc-b4dd-47ea-bd7b-cd3395b524b5","Type":"ContainerStarted","Data":"079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac"}
Jul 03 21:12:29.099308 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:29.099268    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:29.099559 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:29.099528    2553 patch_prober.go:28] interesting pod/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 03 21:12:29.099673 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:29.099611    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 03 21:12:30.100458 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:30.100408    2553 patch_prober.go:28] interesting pod/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 03 21:12:30.100993 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:30.100496    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 03 21:12:31.100297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:31.100258    2553 patch_prober.go:28] interesting pod/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 03 21:12:31.100589 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:31.100558    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 03 21:12:31.766815 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:31.766783    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:31.866588 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:31.866558    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:32.026683 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.026645    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:32.103318 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.103289    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:32.103771 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.103712    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:32.104536 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.104516    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:32.232524 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.232483    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:32.390256 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.390174    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:32.534713 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.534672    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:32.780068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.780040    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:32.878994 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:32.878961    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:33.019314 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:33.019281    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:38.576380 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:38.576353    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:38.679097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:38.679065    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:38.807922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:38.807870    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"]
Jul 03 21:12:38.808174 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:38.808140    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy" containerID="cri-o://079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac" gracePeriod=30
Jul 03 21:12:38.829497 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:38.829414    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:39.172394 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:39.172312    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:39.392859 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:39.392825    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:39.706068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:39.706037    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:39.789220 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:39.789188    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:42.684146 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:42.684119    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:42.770323 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:42.770295    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:42.840828 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:42.840791    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:42.941711 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:42.941624    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:43.229455 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:43.229370    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:43.326878 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:43.326840    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:44.086205 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.086181    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:44.216686 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.216653    2553 generic.go:356] "Generic (PLEG): container finished" podID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerID="079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac" exitCode=0
Jul 03 21:12:44.216847 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.216709    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"
Jul 03 21:12:44.216847 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.216734    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" event={"ID":"e85637cc-b4dd-47ea-bd7b-cd3395b524b5","Type":"ContainerDied","Data":"079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac"}
Jul 03 21:12:44.216847 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.216790    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2" event={"ID":"e85637cc-b4dd-47ea-bd7b-cd3395b524b5","Type":"ContainerDied","Data":"af5d798c3ba97b2792732614fe0af90f83b31db9451a03685ae01a85764fca54"}
Jul 03 21:12:44.216847 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.216818    2553 scope.go:117] "RemoveContainer" containerID="079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac"
Jul 03 21:12:44.225336 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.225316    2553 scope.go:117] "RemoveContainer" containerID="19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"
Jul 03 21:12:44.232536 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.232516    2553 scope.go:117] "RemoveContainer" containerID="079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac"
Jul 03 21:12:44.232825 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:12:44.232807    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac\": container with ID starting with 079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac not found: ID does not exist" containerID="079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac"
Jul 03 21:12:44.232925 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.232837    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac"} err="failed to get container status \"079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac\": rpc error: code = NotFound desc = could not find container \"079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac\": container with ID starting with 079c22f6396f6c59c720d60b99652ddfa21419b7db3539af9a036e4b5c4afdac not found: ID does not exist"
Jul 03 21:12:44.232925 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.232861    2553 scope.go:117] "RemoveContainer" containerID="19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"
Jul 03 21:12:44.233130 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:12:44.233112    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4\": container with ID starting with 19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4 not found: ID does not exist" containerID="19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"
Jul 03 21:12:44.233176 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.233136    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4"} err="failed to get container status \"19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4\": rpc error: code = NotFound desc = could not find container \"19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4\": container with ID starting with 19f4c27cdaf9a1b8c67a01c0facacf4f6ecca4f71938dee7cff962690ef5edf4 not found: ID does not exist"
Jul 03 21:12:44.251177 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251156    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-token\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251303 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251199    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-socket\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251303 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251227    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-data\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251303 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251288    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-certs\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251456 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251312    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istiod-ca-cert\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251456 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251374    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-envoy\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251456 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251413    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-podinfo\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251606 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251493    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:44.251606 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251561    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:44.251606 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251569    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-credential-socket\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251768 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251608    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jd5p\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-kube-api-access-9jd5p\") pod \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\" (UID: \"e85637cc-b4dd-47ea-bd7b-cd3395b524b5\") "
Jul 03 21:12:44.251864 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251743    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 21:12:44.251864 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251819    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:44.252039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251975    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.252039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.251994    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-workload-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.252039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.252006    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istiod-ca-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.252039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.252019    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-credential-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.252231 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.252203    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:44.253447 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.253424    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:12:44.253746 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.253725    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:12:44.253831 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.253812    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 03 21:12:44.253884 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.253861    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-kube-api-access-9jd5p" (OuterVolumeSpecName: "kube-api-access-9jd5p") pod "e85637cc-b4dd-47ea-bd7b-cd3395b524b5" (UID: "e85637cc-b4dd-47ea-bd7b-cd3395b524b5"). InnerVolumeSpecName "kube-api-access-9jd5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:12:44.352836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.352756    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-envoy\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.352836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.352781    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-podinfo\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.352836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.352791    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jd5p\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-kube-api-access-9jd5p\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.352836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.352801    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-token\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.352836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.352808    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85637cc-b4dd-47ea-bd7b-cd3395b524b5-istio-data\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:12:44.541056 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.541028    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"]
Jul 03 21:12:44.543053 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.543022    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-910755-openshift-default-7fb54f4d87-7tth2"]
Jul 03 21:12:44.871163 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:44.871127    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" path="/var/lib/kubelet/pods/e85637cc-b4dd-47ea-bd7b-cd3395b524b5/volumes"
Jul 03 21:12:47.170694 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.170661    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:47.505388 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.505305    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:47.767405 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767320    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"]
Jul 03 21:12:47.767692 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767678    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy"
Jul 03 21:12:47.767738 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767694    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy"
Jul 03 21:12:47.767738 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767705    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy"
Jul 03 21:12:47.767738 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767711    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy"
Jul 03 21:12:47.767738 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767719    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy"
Jul 03 21:12:47.767738 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767725    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc3b979-195e-4a48-b04f-3ea0a2e7598d" containerName="istio-proxy"
Jul 03 21:12:47.767929 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767788    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy"
Jul 03 21:12:47.767929 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.767797    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85637cc-b4dd-47ea-bd7b-cd3395b524b5" containerName="istio-proxy"
Jul 03 21:12:47.771998 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.771976    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.774842 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.774816    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-26641f-openshift-default-dockercfg-2cn8t\""
Jul 03 21:12:47.781711 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.781689    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"]
Jul 03 21:12:47.792977 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.792945    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:47.882518 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882475    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-socket\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882518 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882514    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-token\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882545    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfk2f\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-kube-api-access-lfk2f\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882613    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-certs\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882657    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-credential-socket\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882674    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-data\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882695    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-podinfo\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.882756 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882711    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istiod-ca-cert\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.883015 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.882833    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-envoy\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983423 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983389    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-credential-socket\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983423 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983424    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-data\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983445    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-podinfo\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istiod-ca-cert\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983514    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-envoy\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983560    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-socket\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983583    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-token\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983610    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfk2f\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-kube-api-access-lfk2f\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.983669 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983647    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-certs\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.984098 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.983863    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-data\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.984098 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.984021    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-socket\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.984098 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.984034    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-certs\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.984263 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.984225    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-credential-socket\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.984363 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.984342    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istiod-ca-cert\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.985940 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.985875    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-envoy\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.986096 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.986082    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-podinfo\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.991729 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.991707    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-token\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:47.991832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:47.991776    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfk2f\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-kube-api-access-lfk2f\") pod \"e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") " pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:48.082955 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.082922    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:48.086804 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.086779    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:48.223073 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.223001    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"]
Jul 03 21:12:48.225606 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:12:48.225578    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a79820b_aa2b_4f0d_ab5b_9b6ce2fdd4f7.slice/crio-a6f85b7f6f3ac7abeb49af3becf1029147afc73fe3f8d168ebf332e82c2640d3 WatchSource:0}: Error finding container a6f85b7f6f3ac7abeb49af3becf1029147afc73fe3f8d168ebf332e82c2640d3: Status 404 returned error can't find the container with id a6f85b7f6f3ac7abeb49af3becf1029147afc73fe3f8d168ebf332e82c2640d3
Jul 03 21:12:48.227637 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.227602    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:48.227726 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.227673    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:48.227726 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.227705    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:12:48.236119 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:48.236094    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" event={"ID":"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7","Type":"ContainerStarted","Data":"a6f85b7f6f3ac7abeb49af3becf1029147afc73fe3f8d168ebf332e82c2640d3"}
Jul 03 21:12:49.241606 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:49.241572    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" event={"ID":"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7","Type":"ContainerStarted","Data":"9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"}
Jul 03 21:12:49.262876 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:49.262823    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podStartSLOduration=2.262809507 podStartE2EDuration="2.262809507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:12:49.260542271 +0000 UTC m=+2141.341283678" watchObservedRunningTime="2026-07-03 21:12:49.262809507 +0000 UTC m=+2141.343550914"
Jul 03 21:12:50.083122 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:50.083085    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:50.087868 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:50.087842    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:50.248706 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:50.248673    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:50.249599 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:50.249580    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:12:51.262061 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.262022    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:51.339046 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.339010    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:51.339455 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.339435    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"]
Jul 03 21:12:51.343479 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.343458    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.346043 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.346022    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-26641f-serving-cert\""
Jul 03 21:12:51.346155 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.346135    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-h94dw\""
Jul 03 21:12:51.352745 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.352720    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"]
Jul 03 21:12:51.398428 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.398393    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:51.414481 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.414454    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fq2b\" (UniqueName: \"kubernetes.io/projected/17a9973e-c077-4cde-a8bc-667bdbf62fbb-kube-api-access-7fq2b\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.414622 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.414499    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a9973e-c077-4cde-a8bc-667bdbf62fbb-maas-api-tls\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.414622 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.414557    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a9973e-c077-4cde-a8bc-667bdbf62fbb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.515889 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.515801    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a9973e-c077-4cde-a8bc-667bdbf62fbb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.516063 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.515885    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fq2b\" (UniqueName: \"kubernetes.io/projected/17a9973e-c077-4cde-a8bc-667bdbf62fbb-kube-api-access-7fq2b\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.516063 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.515951    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a9973e-c077-4cde-a8bc-667bdbf62fbb-maas-api-tls\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.516573 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.516550    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a9973e-c077-4cde-a8bc-667bdbf62fbb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.517646 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.517617    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:51.518561 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.518541    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a9973e-c077-4cde-a8bc-667bdbf62fbb-maas-api-tls\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.524614 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.524595    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fq2b\" (UniqueName: \"kubernetes.io/projected/17a9973e-c077-4cde-a8bc-667bdbf62fbb-kube-api-access-7fq2b\") pod \"maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") " pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.658071 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.658033    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:51.793314 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.793281    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"]
Jul 03 21:12:51.796499 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:12:51.796454    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a9973e_c077_4cde_a8bc_667bdbf62fbb.slice/crio-4324d3ead1bdbd9e44038161ba0e4027f51b54db0c9ff46ca83df68a9e85f91a WatchSource:0}: Error finding container 4324d3ead1bdbd9e44038161ba0e4027f51b54db0c9ff46ca83df68a9e85f91a: Status 404 returned error can't find the container with id 4324d3ead1bdbd9e44038161ba0e4027f51b54db0c9ff46ca83df68a9e85f91a
Jul 03 21:12:51.797769 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.797749    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 03 21:12:51.835068 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.835034    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:51.932301 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:51.932270    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:12:52.260924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:52.260809    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5" event={"ID":"17a9973e-c077-4cde-a8bc-667bdbf62fbb","Type":"ContainerStarted","Data":"4324d3ead1bdbd9e44038161ba0e4027f51b54db0c9ff46ca83df68a9e85f91a"}
Jul 03 21:12:54.273275 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:54.273249    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5" event={"ID":"17a9973e-c077-4cde-a8bc-667bdbf62fbb","Type":"ContainerStarted","Data":"5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298"}
Jul 03 21:12:54.273641 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:54.273351    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:12:54.290340 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:54.290294    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5" podStartSLOduration=1.101852644 podStartE2EDuration="3.290280444s" totalImagesPullingTime="2.1884278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:12:54.288941297 +0000 UTC m=+2146.369682707" watchObservedRunningTime="2026-07-03 21:12:54.290280444 +0000 UTC m=+2146.371021851"
Jul 03 21:12:55.752130 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:12:55.752086    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:00.284145 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:00.284067    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:13:02.693047 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:02.693012    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:02.795383 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:02.795351    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:03.836807 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:03.836767    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:03.937462 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:03.937419    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:05.261259 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.259930    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"]
Jul 03 21:13:05.265336 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.265255    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.267823 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.267699    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"]
Jul 03 21:13:05.268534 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.268514    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1a4753\"/\"openshift-service-ca.crt\""
Jul 03 21:13:05.271517 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.271498    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1a4753\"/\"default-dockercfg-4bhdn\""
Jul 03 21:13:05.272148 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.272130    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1a4753\"/\"auth-test-model-1a4753-kserve-self-signed-certs\""
Jul 03 21:13:05.272522 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.272495    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1a4753\"/\"kube-root-ca.crt\""
Jul 03 21:13:05.274825 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.274803    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:05.358137 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.358104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgs2\" (UniqueName: \"kubernetes.io/projected/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-kube-api-access-kfgs2\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.358316 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.358142    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-home\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.358316 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.358175    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tmp-dir\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.358316 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.358250    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-dshm\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.358316 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.358285    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tls-certs\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.358316 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.358308    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-model-cache\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.458823 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.458780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgs2\" (UniqueName: \"kubernetes.io/projected/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-kube-api-access-kfgs2\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.458823 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.458826    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-home\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459120 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.458863    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tmp-dir\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459120 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.458946    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-dshm\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459120 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.458977    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tls-certs\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459120 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.459005    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-model-cache\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459304 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.459281    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tmp-dir\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459441 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.459411    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-home\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.459518 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.459505    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-model-cache\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.461361 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.461338    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-dshm\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.461738 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.461715    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tls-certs\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.467799 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.467767    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgs2\" (UniqueName: \"kubernetes.io/projected/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-kube-api-access-kfgs2\") pod \"auth-test-model-1a4753-kserve-66567f547-wfpzh\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") " pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.579540 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.579503    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:05.712590 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:05.712562    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"]
Jul 03 21:13:05.714655 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:13:05.714629    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2662838_e04b_4c63_8e1a_aa3e52fd42e7.slice/crio-2848cb0cabe6df570feb96588c1855c8c046fc7621022bc2cd16327e0bba1eb7 WatchSource:0}: Error finding container 2848cb0cabe6df570feb96588c1855c8c046fc7621022bc2cd16327e0bba1eb7: Status 404 returned error can't find the container with id 2848cb0cabe6df570feb96588c1855c8c046fc7621022bc2cd16327e0bba1eb7
Jul 03 21:13:06.291081 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:06.291042    2553 patch_prober.go:28] interesting pod/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 03 21:13:06.291471 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:06.291103    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 03 21:13:06.343213 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:06.343171    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh" event={"ID":"e2662838-e04b-4c63-8e1a-aa3e52fd42e7","Type":"ContainerStarted","Data":"b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d"}
Jul 03 21:13:06.343213 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:06.343220    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh" event={"ID":"e2662838-e04b-4c63-8e1a-aa3e52fd42e7","Type":"ContainerStarted","Data":"2848cb0cabe6df570feb96588c1855c8c046fc7621022bc2cd16327e0bba1eb7"}
Jul 03 21:13:06.362587 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:06.362529    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh" podStartSLOduration=1.362505942 podStartE2EDuration="1.362505942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:13:06.359963831 +0000 UTC m=+2158.440705239" watchObservedRunningTime="2026-07-03 21:13:06.362505942 +0000 UTC m=+2158.443247353"
Jul 03 21:13:07.349508 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:07.349478    2553 generic.go:356] "Generic (PLEG): container finished" podID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerID="9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5" exitCode=0
Jul 03 21:13:07.350014 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:07.349554    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" event={"ID":"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7","Type":"ContainerDied","Data":"9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"}
Jul 03 21:13:07.350066 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:07.350035    2553 scope.go:117] "RemoveContainer" containerID="9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"
Jul 03 21:13:07.350982 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:07.350953    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:13:07.351098 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:07.351037    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:13:07.351098 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:07.351085    2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 03 21:13:08.357659 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:08.357621    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" event={"ID":"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7","Type":"ContainerStarted","Data":"203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51"}
Jul 03 21:13:09.083114 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:09.083082    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:13:09.083342 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:09.083323    2553 patch_prober.go:28] interesting pod/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 03 21:13:09.083387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:09.083370    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 03 21:13:10.083293 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:10.083251    2553 patch_prober.go:28] interesting pod/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 03 21:13:10.083769 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:10.083325    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 03 21:13:11.083356 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:11.083323    2553 patch_prober.go:28] interesting pod/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 03 21:13:11.083719 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:11.083370    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 03 21:13:12.083957 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:12.083858    2553 patch_prober.go:28] interesting pod/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 03 21:13:12.083957 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:12.083935    2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 03 21:13:12.568814 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:12.568776    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:12.668090 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:12.668057    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:13:13.087030 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:13.087003    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:13:13.087410 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:13.087240    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:13:13.087860 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:13.087842    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:13:15.580294 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:15.580260    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:15.580682 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:15.580524    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:15.592772 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:15.592749    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:16.407355 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:16.407325    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:13:38.782201 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:13:38.782168    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:15:09.102836 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:09.102795    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:15:28.584475 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:28.584439    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"]
Jul 03 21:15:28.603780 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:28.603741    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:15:29.549205 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:29.549164    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:15:31.451760 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.451729    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"]
Jul 03 21:15:31.455318 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.455301    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.457866 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.457847    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1a4753\"/\"sub-test-model-1a4753-kserve-self-signed-certs\""
Jul 03 21:15:31.463324 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.463298    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"]
Jul 03 21:15:31.475991 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.475961    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:15:31.558337 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.558304    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-tmp-dir\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.558508 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.558355    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-home\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.558508 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.558414    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmpg\" (UniqueName: \"kubernetes.io/projected/96060709-c3d4-4baf-8893-b6bdae149888-kube-api-access-njmpg\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.558508 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.558453    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-dshm\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.558631 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.558508    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-model-cache\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.558631 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.558541    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96060709-c3d4-4baf-8893-b6bdae149888-tls-certs\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659397 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659358    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-home\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659397 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659393    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmpg\" (UniqueName: \"kubernetes.io/projected/96060709-c3d4-4baf-8893-b6bdae149888-kube-api-access-njmpg\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659659 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659417    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-dshm\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659659 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-model-cache\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659659 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96060709-c3d4-4baf-8893-b6bdae149888-tls-certs\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659659 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659574    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-tmp-dir\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659865 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659835    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-home\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.659945 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.659929    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-model-cache\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.660162 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.660140    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-tmp-dir\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.662100 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.662074    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96060709-c3d4-4baf-8893-b6bdae149888-tls-certs\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.662305 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.662290    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-dshm\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.667325 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.667304    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmpg\" (UniqueName: \"kubernetes.io/projected/96060709-c3d4-4baf-8893-b6bdae149888-kube-api-access-njmpg\") pod \"sub-test-model-1a4753-kserve-68577d8965-2wr8f\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") " pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.765360 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.765272    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:31.900343 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:31.900305    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"]
Jul 03 21:15:31.903300 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:15:31.903273    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96060709_c3d4_4baf_8893_b6bdae149888.slice/crio-7f1031156881fe7a1b3c91c63b2338016932e2e742d7e6589b71507d0e6b7f7e WatchSource:0}: Error finding container 7f1031156881fe7a1b3c91c63b2338016932e2e742d7e6589b71507d0e6b7f7e: Status 404 returned error can't find the container with id 7f1031156881fe7a1b3c91c63b2338016932e2e742d7e6589b71507d0e6b7f7e
Jul 03 21:15:32.055028 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:32.054994    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f" event={"ID":"96060709-c3d4-4baf-8893-b6bdae149888","Type":"ContainerStarted","Data":"94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8"}
Jul 03 21:15:32.055203 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:32.055044    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f" event={"ID":"96060709-c3d4-4baf-8893-b6bdae149888","Type":"ContainerStarted","Data":"7f1031156881fe7a1b3c91c63b2338016932e2e742d7e6589b71507d0e6b7f7e"}
Jul 03 21:15:32.071981 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:32.071936    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f" podStartSLOduration=1.071921035 podStartE2EDuration="1.071921035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:15:32.070227402 +0000 UTC m=+2304.150968802" watchObservedRunningTime="2026-07-03 21:15:32.071921035 +0000 UTC m=+2304.152662442"
Jul 03 21:15:41.765975 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:41.765940    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:41.766516 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:41.766016    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:41.778507 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:41.778486    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:42.116287 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:42.116260    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:15:43.606301 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.606258    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh" podUID="e2662838-e04b-4c63-8e1a-aa3e52fd42e7" containerName="main" containerID="cri-o://b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d" gracePeriod=45
Jul 03 21:15:43.862143 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.862090    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:15:43.974229 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974193    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tmp-dir\") pod \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") "
Jul 03 21:15:43.974418 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974249    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-dshm\") pod \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") "
Jul 03 21:15:43.974418 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974282    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tls-certs\") pod \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") "
Jul 03 21:15:43.974418 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974362    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-model-cache\") pod \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") "
Jul 03 21:15:43.974418 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974392    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgs2\" (UniqueName: \"kubernetes.io/projected/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-kube-api-access-kfgs2\") pod \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") "
Jul 03 21:15:43.974638 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974436    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-home\") pod \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\" (UID: \"e2662838-e04b-4c63-8e1a-aa3e52fd42e7\") "
Jul 03 21:15:43.974638 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974526    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2662838-e04b-4c63-8e1a-aa3e52fd42e7" (UID: "e2662838-e04b-4c63-8e1a-aa3e52fd42e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:15:43.974766 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974716    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2662838-e04b-4c63-8e1a-aa3e52fd42e7" (UID: "e2662838-e04b-4c63-8e1a-aa3e52fd42e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:15:43.974819 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974788    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tmp-dir\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:15:43.974865 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.974812    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-home" (OuterVolumeSpecName: "home") pod "e2662838-e04b-4c63-8e1a-aa3e52fd42e7" (UID: "e2662838-e04b-4c63-8e1a-aa3e52fd42e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:15:43.976560 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.976530    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-dshm" (OuterVolumeSpecName: "dshm") pod "e2662838-e04b-4c63-8e1a-aa3e52fd42e7" (UID: "e2662838-e04b-4c63-8e1a-aa3e52fd42e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:15:43.976665 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.976596    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2662838-e04b-4c63-8e1a-aa3e52fd42e7" (UID: "e2662838-e04b-4c63-8e1a-aa3e52fd42e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:15:43.976665 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:43.976629    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-kube-api-access-kfgs2" (OuterVolumeSpecName: "kube-api-access-kfgs2") pod "e2662838-e04b-4c63-8e1a-aa3e52fd42e7" (UID: "e2662838-e04b-4c63-8e1a-aa3e52fd42e7"). InnerVolumeSpecName "kube-api-access-kfgs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:15:44.076162 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.076121    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-tls-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:15:44.076162 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.076155    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-model-cache\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:15:44.076162 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.076164    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgs2\" (UniqueName: \"kubernetes.io/projected/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-kube-api-access-kfgs2\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:15:44.076162 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.076172    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-home\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:15:44.076432 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.076181    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2662838-e04b-4c63-8e1a-aa3e52fd42e7-dshm\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:15:44.116259 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.116170    2553 generic.go:356] "Generic (PLEG): container finished" podID="e2662838-e04b-4c63-8e1a-aa3e52fd42e7" containerID="b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d" exitCode=0
Jul 03 21:15:44.116259 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.116212    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"
Jul 03 21:15:44.116475 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.116256    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh" event={"ID":"e2662838-e04b-4c63-8e1a-aa3e52fd42e7","Type":"ContainerDied","Data":"b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d"}
Jul 03 21:15:44.116475 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.116292    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh" event={"ID":"e2662838-e04b-4c63-8e1a-aa3e52fd42e7","Type":"ContainerDied","Data":"2848cb0cabe6df570feb96588c1855c8c046fc7621022bc2cd16327e0bba1eb7"}
Jul 03 21:15:44.116475 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.116310    2553 scope.go:117] "RemoveContainer" containerID="b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d"
Jul 03 21:15:44.125312 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.125297    2553 scope.go:117] "RemoveContainer" containerID="b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d"
Jul 03 21:15:44.125561 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:15:44.125545    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d\": container with ID starting with b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d not found: ID does not exist" containerID="b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d"
Jul 03 21:15:44.125626 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.125568    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d"} err="failed to get container status \"b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d\": rpc error: code = NotFound desc = could not find container \"b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d\": container with ID starting with b4dad755dd5bf580e2a48f8c9df41dbad3b22f2ceca3d6400836ba22a09e433d not found: ID does not exist"
Jul 03 21:15:44.137302 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.137274    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"]
Jul 03 21:15:44.140040 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.140019    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/auth-test-model-1a4753-kserve-66567f547-wfpzh"]
Jul 03 21:15:44.876820 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:15:44.876781    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2662838-e04b-4c63-8e1a-aa3e52fd42e7" path="/var/lib/kubelet/pods/e2662838-e04b-4c63-8e1a-aa3e52fd42e7/volumes"
Jul 03 21:16:00.052744 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:16:00.052664    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:16:20.529168 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:16:20.529131    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:16:31.852065 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:16:31.852021    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:01.597884 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:01.597849    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"]
Jul 03 21:17:01.612500 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:01.612467    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:02.586017 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:02.585979    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:04.440151 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:04.440113    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:04.490207 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:04.490162    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:09.755813 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.755780    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"]
Jul 03 21:17:09.756194 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.756169    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2662838-e04b-4c63-8e1a-aa3e52fd42e7" containerName="main"
Jul 03 21:17:09.756194 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.756183    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2662838-e04b-4c63-8e1a-aa3e52fd42e7" containerName="main"
Jul 03 21:17:09.756267 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.756251    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2662838-e04b-4c63-8e1a-aa3e52fd42e7" containerName="main"
Jul 03 21:17:09.759500 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.759476    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.762339 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.762316    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1a4753\"/\"rate-test-model-1a4753-kserve-self-signed-certs\""
Jul 03 21:17:09.767132 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.767108    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"]
Jul 03 21:17:09.769661 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.769553    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:09.831197 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.831166    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-dshm\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.831395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.831206    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c9ea3-86a6-48ea-8133-f15467c9d642-tls-certs\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.831395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.831327    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-tmp-dir\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.831395 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.831373    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-home\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.831571 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.831410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-model-cache\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.831571 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.831458    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffwh\" (UniqueName: \"kubernetes.io/projected/e69c9ea3-86a6-48ea-8133-f15467c9d642-kube-api-access-6ffwh\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-tmp-dir\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932387 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932390    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-home\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932671 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932413    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-model-cache\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932671 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932438    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffwh\" (UniqueName: \"kubernetes.io/projected/e69c9ea3-86a6-48ea-8133-f15467c9d642-kube-api-access-6ffwh\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932671 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932534    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-dshm\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932671 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932604    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c9ea3-86a6-48ea-8133-f15467c9d642-tls-certs\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.932933 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.932841    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-home\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.933039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.933013    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-model-cache\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.933039 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.933032    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-tmp-dir\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.934826 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.934804    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-dshm\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.935095 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.935078    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c9ea3-86a6-48ea-8133-f15467c9d642-tls-certs\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:09.939855 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:09.939833    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffwh\" (UniqueName: \"kubernetes.io/projected/e69c9ea3-86a6-48ea-8133-f15467c9d642-kube-api-access-6ffwh\") pod \"rate-test-model-1a4753-kserve-55c7b5cd9f-mq296\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") " pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:10.072471 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:10.072439    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:10.205258 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:10.205226    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"]
Jul 03 21:17:10.208785 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:17:10.208756    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode69c9ea3_86a6_48ea_8133_f15467c9d642.slice/crio-1acd2df49f8a852eb3fb01e34641539a19de8b9bcbccb334b9b291bfa4f9dfd9 WatchSource:0}: Error finding container 1acd2df49f8a852eb3fb01e34641539a19de8b9bcbccb334b9b291bfa4f9dfd9: Status 404 returned error can't find the container with id 1acd2df49f8a852eb3fb01e34641539a19de8b9bcbccb334b9b291bfa4f9dfd9
Jul 03 21:17:10.523117 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:10.523084    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296" event={"ID":"e69c9ea3-86a6-48ea-8133-f15467c9d642","Type":"ContainerStarted","Data":"9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33"}
Jul 03 21:17:10.523307 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:10.523130    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296" event={"ID":"e69c9ea3-86a6-48ea-8133-f15467c9d642","Type":"ContainerStarted","Data":"1acd2df49f8a852eb3fb01e34641539a19de8b9bcbccb334b9b291bfa4f9dfd9"}
Jul 03 21:17:10.545245 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:10.545193    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296" podStartSLOduration=1.545171184 podStartE2EDuration="1.545171184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:17:10.543764957 +0000 UTC m=+2402.624506365" watchObservedRunningTime="2026-07-03 21:17:10.545171184 +0000 UTC m=+2402.625912592"
Jul 03 21:17:16.621965 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.621919    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f" podUID="96060709-c3d4-4baf-8893-b6bdae149888" containerName="main" containerID="cri-o://94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8" gracePeriod=45
Jul 03 21:17:16.882215 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.882159    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:17:16.997172 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997134    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-dshm\") pod \"96060709-c3d4-4baf-8893-b6bdae149888\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") "
Jul 03 21:17:16.997381 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997271    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-home\") pod \"96060709-c3d4-4baf-8893-b6bdae149888\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") "
Jul 03 21:17:16.997381 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997316    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-model-cache\") pod \"96060709-c3d4-4baf-8893-b6bdae149888\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") "
Jul 03 21:17:16.997381 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997342    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96060709-c3d4-4baf-8893-b6bdae149888-tls-certs\") pod \"96060709-c3d4-4baf-8893-b6bdae149888\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") "
Jul 03 21:17:16.997541 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997381    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njmpg\" (UniqueName: \"kubernetes.io/projected/96060709-c3d4-4baf-8893-b6bdae149888-kube-api-access-njmpg\") pod \"96060709-c3d4-4baf-8893-b6bdae149888\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") "
Jul 03 21:17:16.997541 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997413    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-tmp-dir\") pod \"96060709-c3d4-4baf-8893-b6bdae149888\" (UID: \"96060709-c3d4-4baf-8893-b6bdae149888\") "
Jul 03 21:17:16.997648 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997566    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-home" (OuterVolumeSpecName: "home") pod "96060709-c3d4-4baf-8893-b6bdae149888" (UID: "96060709-c3d4-4baf-8893-b6bdae149888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:17:16.997702 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997664    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-home\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:17:16.997832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997787    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-model-cache" (OuterVolumeSpecName: "model-cache") pod "96060709-c3d4-4baf-8893-b6bdae149888" (UID: "96060709-c3d4-4baf-8893-b6bdae149888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:17:16.997832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.997801    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96060709-c3d4-4baf-8893-b6bdae149888" (UID: "96060709-c3d4-4baf-8893-b6bdae149888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:17:16.999874 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:16.999843    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-dshm" (OuterVolumeSpecName: "dshm") pod "96060709-c3d4-4baf-8893-b6bdae149888" (UID: "96060709-c3d4-4baf-8893-b6bdae149888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:17:17.000080 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.000064    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96060709-c3d4-4baf-8893-b6bdae149888-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96060709-c3d4-4baf-8893-b6bdae149888" (UID: "96060709-c3d4-4baf-8893-b6bdae149888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:17:17.000239 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.000215    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96060709-c3d4-4baf-8893-b6bdae149888-kube-api-access-njmpg" (OuterVolumeSpecName: "kube-api-access-njmpg") pod "96060709-c3d4-4baf-8893-b6bdae149888" (UID: "96060709-c3d4-4baf-8893-b6bdae149888"). InnerVolumeSpecName "kube-api-access-njmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:17:17.098284 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.098248    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-model-cache\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:17:17.098284 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.098273    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96060709-c3d4-4baf-8893-b6bdae149888-tls-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:17:17.098284 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.098282    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njmpg\" (UniqueName: \"kubernetes.io/projected/96060709-c3d4-4baf-8893-b6bdae149888-kube-api-access-njmpg\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:17:17.098284 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.098293    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-tmp-dir\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:17:17.098612 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.098301    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96060709-c3d4-4baf-8893-b6bdae149888-dshm\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:17:17.559419 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.559392    2553 generic.go:356] "Generic (PLEG): container finished" podID="96060709-c3d4-4baf-8893-b6bdae149888" containerID="94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8" exitCode=0
Jul 03 21:17:17.559612 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.559434    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"
Jul 03 21:17:17.559612 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.559480    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f" event={"ID":"96060709-c3d4-4baf-8893-b6bdae149888","Type":"ContainerDied","Data":"94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8"}
Jul 03 21:17:17.559612 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.559524    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f" event={"ID":"96060709-c3d4-4baf-8893-b6bdae149888","Type":"ContainerDied","Data":"7f1031156881fe7a1b3c91c63b2338016932e2e742d7e6589b71507d0e6b7f7e"}
Jul 03 21:17:17.559612 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.559542    2553 scope.go:117] "RemoveContainer" containerID="94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8"
Jul 03 21:17:17.574133 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.574108    2553 scope.go:117] "RemoveContainer" containerID="94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8"
Jul 03 21:17:17.574425 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:17:17.574403    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8\": container with ID starting with 94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8 not found: ID does not exist" containerID="94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8"
Jul 03 21:17:17.574499 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.574433    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8"} err="failed to get container status \"94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8\": rpc error: code = NotFound desc = could not find container \"94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8\": container with ID starting with 94b6062fd52a4115f20cf7567a27af579caeb57e28a86eee84d07eb3a48f26c8 not found: ID does not exist"
Jul 03 21:17:17.583564 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.583539    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"]
Jul 03 21:17:17.585238 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:17.585222    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/sub-test-model-1a4753-kserve-68577d8965-2wr8f"]
Jul 03 21:17:18.871508 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:18.871468    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96060709-c3d4-4baf-8893-b6bdae149888" path="/var/lib/kubelet/pods/96060709-c3d4-4baf-8893-b6bdae149888/volumes"
Jul 03 21:17:20.072749 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:20.072719    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:20.072749 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:20.072759    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:20.085254 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:20.085231    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:20.586631 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:20.586605    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:17:27.747359 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.747324    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"]
Jul 03 21:17:27.747733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.747695    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96060709-c3d4-4baf-8893-b6bdae149888" containerName="main"
Jul 03 21:17:27.747733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.747706    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="96060709-c3d4-4baf-8893-b6bdae149888" containerName="main"
Jul 03 21:17:27.747804 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.747756    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="96060709-c3d4-4baf-8893-b6bdae149888" containerName="main"
Jul 03 21:17:27.752124 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.752104    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.756300 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.756271    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26641f\"/\"openshift-service-ca.crt\""
Jul 03 21:17:27.756577 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.756553    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-26641f\"/\"kube-root-ca.crt\""
Jul 03 21:17:27.756699 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.756551    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26641f\"/\"default-dockercfg-hpf6f\""
Jul 03 21:17:27.756765 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.756733    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-26641f\"/\"rate-test-model-26641f-kserve-self-signed-certs\""
Jul 03 21:17:27.758557 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.758534    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"]
Jul 03 21:17:27.861985 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.861945    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:17:27.892174 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.892144    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-home\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.892315 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.892194    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-dshm\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.892315 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.892243    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtgf\" (UniqueName: \"kubernetes.io/projected/70ee111e-d404-4b4e-ab85-c8971b4b06f3-kube-api-access-svtgf\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.892315 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.892263    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-model-cache\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.892315 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.892292    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tmp-dir\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.892461 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.892328    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tls-certs\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.993718 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.993682    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-dshm\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.993927 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.993728    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtgf\" (UniqueName: \"kubernetes.io/projected/70ee111e-d404-4b4e-ab85-c8971b4b06f3-kube-api-access-svtgf\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.993927 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.993851    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-model-cache\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.994404 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.994234    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tmp-dir\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.994520 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.994472    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tls-certs\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.994582 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.994558    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-home\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.994646 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.994618    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-model-cache\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.994705 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.994649    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tmp-dir\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.999965 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.994876    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-home\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:27.999965 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:27.998122    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-dshm\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:28.002813 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.002786    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tls-certs\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:28.003977 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.003952    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtgf\" (UniqueName: \"kubernetes.io/projected/70ee111e-d404-4b4e-ab85-c8971b4b06f3-kube-api-access-svtgf\") pod \"rate-test-model-26641f-kserve-59f8f4dc7f-tppmq\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") " pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:28.061566 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.061533    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:28.200355 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.200329    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"]
Jul 03 21:17:28.202789 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:17:28.202759    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ee111e_d404_4b4e_ab85_c8971b4b06f3.slice/crio-f083dd06186e932bca3a3d154e0013023d0a35de16bb8c60bc14291f2ad00bcb WatchSource:0}: Error finding container f083dd06186e932bca3a3d154e0013023d0a35de16bb8c60bc14291f2ad00bcb: Status 404 returned error can't find the container with id f083dd06186e932bca3a3d154e0013023d0a35de16bb8c60bc14291f2ad00bcb
Jul 03 21:17:28.614956 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.614918    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq" event={"ID":"70ee111e-d404-4b4e-ab85-c8971b4b06f3","Type":"ContainerStarted","Data":"c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92"}
Jul 03 21:17:28.615124 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.614969    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq" event={"ID":"70ee111e-d404-4b4e-ab85-c8971b4b06f3","Type":"ContainerStarted","Data":"f083dd06186e932bca3a3d154e0013023d0a35de16bb8c60bc14291f2ad00bcb"}
Jul 03 21:17:28.631347 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:28.631300    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq" podStartSLOduration=1.6312851720000001 podStartE2EDuration="1.631285172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:17:28.630019351 +0000 UTC m=+2420.710760759" watchObservedRunningTime="2026-07-03 21:17:28.631285172 +0000 UTC m=+2420.712026580"
Jul 03 21:17:38.062334 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:38.062249    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:38.062334 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:38.062304    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:38.074826 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:38.074799    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:38.674746 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:38.674719    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:17:42.637470 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:17:42.637430    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:03.422362 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:03.422327    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:03.517241 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:03.517201    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:04.560795 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:04.560758    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:04.661772 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:04.661733    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:08.076503 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:08.076470    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"]
Jul 03 21:18:08.089792 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:08.089760    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:08.931190 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:08.931112    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"]
Jul 03 21:18:08.948135 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:08.948097    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:09.957103 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:09.957021    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:10.219666 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:10.219579    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:10.348719 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:10.348687    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:13.176573 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:13.176541    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:13.277414 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:13.277377    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:15.421830 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:15.421799    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:23.096832 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.096770    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296" podUID="e69c9ea3-86a6-48ea-8133-f15467c9d642" containerName="main" containerID="cri-o://9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33" gracePeriod=45
Jul 03 21:18:23.353554 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.353494    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:18:23.510781 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.510745    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffwh\" (UniqueName: \"kubernetes.io/projected/e69c9ea3-86a6-48ea-8133-f15467c9d642-kube-api-access-6ffwh\") pod \"e69c9ea3-86a6-48ea-8133-f15467c9d642\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") "
Jul 03 21:18:23.510989 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.510792    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-model-cache\") pod \"e69c9ea3-86a6-48ea-8133-f15467c9d642\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") "
Jul 03 21:18:23.510989 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.510828    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-dshm\") pod \"e69c9ea3-86a6-48ea-8133-f15467c9d642\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") "
Jul 03 21:18:23.510989 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.510859    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-tmp-dir\") pod \"e69c9ea3-86a6-48ea-8133-f15467c9d642\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") "
Jul 03 21:18:23.511159 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.510979    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-home\") pod \"e69c9ea3-86a6-48ea-8133-f15467c9d642\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") "
Jul 03 21:18:23.511159 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511091    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c9ea3-86a6-48ea-8133-f15467c9d642-tls-certs\") pod \"e69c9ea3-86a6-48ea-8133-f15467c9d642\" (UID: \"e69c9ea3-86a6-48ea-8133-f15467c9d642\") "
Jul 03 21:18:23.511159 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511116    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e69c9ea3-86a6-48ea-8133-f15467c9d642" (UID: "e69c9ea3-86a6-48ea-8133-f15467c9d642"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:23.511159 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511128    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-model-cache" (OuterVolumeSpecName: "model-cache") pod "e69c9ea3-86a6-48ea-8133-f15467c9d642" (UID: "e69c9ea3-86a6-48ea-8133-f15467c9d642"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:23.511410 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511265    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-home" (OuterVolumeSpecName: "home") pod "e69c9ea3-86a6-48ea-8133-f15467c9d642" (UID: "e69c9ea3-86a6-48ea-8133-f15467c9d642"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:23.511537 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511512    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-model-cache\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:23.511537 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511539    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-tmp-dir\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:23.511735 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.511549    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-home\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:23.512954 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.512933    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-dshm" (OuterVolumeSpecName: "dshm") pod "e69c9ea3-86a6-48ea-8133-f15467c9d642" (UID: "e69c9ea3-86a6-48ea-8133-f15467c9d642"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:23.513049 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.513013    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e69c9ea3-86a6-48ea-8133-f15467c9d642-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e69c9ea3-86a6-48ea-8133-f15467c9d642" (UID: "e69c9ea3-86a6-48ea-8133-f15467c9d642"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:18:23.513049 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.513018    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69c9ea3-86a6-48ea-8133-f15467c9d642-kube-api-access-6ffwh" (OuterVolumeSpecName: "kube-api-access-6ffwh") pod "e69c9ea3-86a6-48ea-8133-f15467c9d642" (UID: "e69c9ea3-86a6-48ea-8133-f15467c9d642"). InnerVolumeSpecName "kube-api-access-6ffwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:18:23.612724 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.612640    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e69c9ea3-86a6-48ea-8133-f15467c9d642-tls-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:23.612724 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.612670    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffwh\" (UniqueName: \"kubernetes.io/projected/e69c9ea3-86a6-48ea-8133-f15467c9d642-kube-api-access-6ffwh\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:23.612724 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.612679    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e69c9ea3-86a6-48ea-8133-f15467c9d642-dshm\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:23.886081 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.885989    2553 generic.go:356] "Generic (PLEG): container finished" podID="e69c9ea3-86a6-48ea-8133-f15467c9d642" containerID="9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33" exitCode=0
Jul 03 21:18:23.886081 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.886030    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"
Jul 03 21:18:23.886081 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.886061    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296" event={"ID":"e69c9ea3-86a6-48ea-8133-f15467c9d642","Type":"ContainerDied","Data":"9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33"}
Jul 03 21:18:23.886338 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.886103    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296" event={"ID":"e69c9ea3-86a6-48ea-8133-f15467c9d642","Type":"ContainerDied","Data":"1acd2df49f8a852eb3fb01e34641539a19de8b9bcbccb334b9b291bfa4f9dfd9"}
Jul 03 21:18:23.886338 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.886122    2553 scope.go:117] "RemoveContainer" containerID="9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33"
Jul 03 21:18:23.895625 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.895606    2553 scope.go:117] "RemoveContainer" containerID="9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33"
Jul 03 21:18:23.895924 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:18:23.895883    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33\": container with ID starting with 9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33 not found: ID does not exist" containerID="9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33"
Jul 03 21:18:23.896026 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.895932    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33"} err="failed to get container status \"9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33\": rpc error: code = NotFound desc = could not find container \"9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33\": container with ID starting with 9f3f7754a4394e4f575f187ef2dd2d4314f14c0e5e2ef961775c28a5beb41e33 not found: ID does not exist"
Jul 03 21:18:23.907717 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.907694    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"]
Jul 03 21:18:23.909742 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.909723    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1a4753/rate-test-model-1a4753-kserve-55c7b5cd9f-mq296"]
Jul 03 21:18:23.952336 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:23.952312    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq" podUID="70ee111e-d404-4b4e-ab85-c8971b4b06f3" containerName="main" containerID="cri-o://c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92" gracePeriod=45
Jul 03 21:18:24.203582 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.203559    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:18:24.318317 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318287    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-home\") pod \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") "
Jul 03 21:18:24.318317 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318323    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tls-certs\") pod \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") "
Jul 03 21:18:24.318570 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318386    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-dshm\") pod \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") "
Jul 03 21:18:24.318570 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318406    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svtgf\" (UniqueName: \"kubernetes.io/projected/70ee111e-d404-4b4e-ab85-c8971b4b06f3-kube-api-access-svtgf\") pod \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") "
Jul 03 21:18:24.318570 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318434    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tmp-dir\") pod \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") "
Jul 03 21:18:24.318734 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318573    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-model-cache\") pod \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\" (UID: \"70ee111e-d404-4b4e-ab85-c8971b4b06f3\") "
Jul 03 21:18:24.318734 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318635    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-home" (OuterVolumeSpecName: "home") pod "70ee111e-d404-4b4e-ab85-c8971b4b06f3" (UID: "70ee111e-d404-4b4e-ab85-c8971b4b06f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:24.318734 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318719    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70ee111e-d404-4b4e-ab85-c8971b4b06f3" (UID: "70ee111e-d404-4b4e-ab85-c8971b4b06f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:24.318861 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318810    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "70ee111e-d404-4b4e-ab85-c8971b4b06f3" (UID: "70ee111e-d404-4b4e-ab85-c8971b4b06f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:24.318936 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318887    2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tmp-dir\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:24.318993 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318944    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-model-cache\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:24.318993 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.318957    2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-home\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:24.320702 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.320683    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ee111e-d404-4b4e-ab85-c8971b4b06f3-kube-api-access-svtgf" (OuterVolumeSpecName: "kube-api-access-svtgf") pod "70ee111e-d404-4b4e-ab85-c8971b4b06f3" (UID: "70ee111e-d404-4b4e-ab85-c8971b4b06f3"). InnerVolumeSpecName "kube-api-access-svtgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:18:24.320928 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.320876    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-dshm" (OuterVolumeSpecName: "dshm") pod "70ee111e-d404-4b4e-ab85-c8971b4b06f3" (UID: "70ee111e-d404-4b4e-ab85-c8971b4b06f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:18:24.320928 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.320919    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70ee111e-d404-4b4e-ab85-c8971b4b06f3" (UID: "70ee111e-d404-4b4e-ab85-c8971b4b06f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:18:24.419379 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.419348    2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ee111e-d404-4b4e-ab85-c8971b4b06f3-tls-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:24.419379 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.419374    2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ee111e-d404-4b4e-ab85-c8971b4b06f3-dshm\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:24.419379 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.419383    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtgf\" (UniqueName: \"kubernetes.io/projected/70ee111e-d404-4b4e-ab85-c8971b4b06f3-kube-api-access-svtgf\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:24.871463 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.871429    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69c9ea3-86a6-48ea-8133-f15467c9d642" path="/var/lib/kubelet/pods/e69c9ea3-86a6-48ea-8133-f15467c9d642/volumes"
Jul 03 21:18:24.891884 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.891857    2553 generic.go:356] "Generic (PLEG): container finished" podID="70ee111e-d404-4b4e-ab85-c8971b4b06f3" containerID="c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92" exitCode=0
Jul 03 21:18:24.892008 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.891933    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"
Jul 03 21:18:24.892057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.891932    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq" event={"ID":"70ee111e-d404-4b4e-ab85-c8971b4b06f3","Type":"ContainerDied","Data":"c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92"}
Jul 03 21:18:24.892057 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.892041    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq" event={"ID":"70ee111e-d404-4b4e-ab85-c8971b4b06f3","Type":"ContainerDied","Data":"f083dd06186e932bca3a3d154e0013023d0a35de16bb8c60bc14291f2ad00bcb"}
Jul 03 21:18:24.892152 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.892059    2553 scope.go:117] "RemoveContainer" containerID="c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92"
Jul 03 21:18:24.900603 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.900583    2553 scope.go:117] "RemoveContainer" containerID="c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92"
Jul 03 21:18:24.900845 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:18:24.900829    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92\": container with ID starting with c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92 not found: ID does not exist" containerID="c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92"
Jul 03 21:18:24.900912 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.900862    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92"} err="failed to get container status \"c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92\": rpc error: code = NotFound desc = could not find container \"c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92\": container with ID starting with c6be137ae12980c843ddb9582a767c904d786a5d807400207e3916e3ff10ef92 not found: ID does not exist"
Jul 03 21:18:24.907605 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.907585    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"]
Jul 03 21:18:24.909247 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:24.909217    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-26641f/rate-test-model-26641f-kserve-59f8f4dc7f-tppmq"]
Jul 03 21:18:26.870211 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:26.870169    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ee111e-d404-4b4e-ab85-c8971b4b06f3" path="/var/lib/kubelet/pods/70ee111e-d404-4b4e-ab85-c8971b4b06f3/volumes"
Jul 03 21:18:47.147118 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:47.147075    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:52.162878 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:52.162842    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:52.429515 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:52.429425    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:52.662442 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:52.662405    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:53.003542 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.003508    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:53.298851 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.298820    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"]
Jul 03 21:18:53.299603 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.299570    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5" podUID="17a9973e-c077-4cde-a8bc-667bdbf62fbb" containerName="maas-api" containerID="cri-o://5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298" gracePeriod=30
Jul 03 21:18:53.323281 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.323242    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:53.423161 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.423116    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:53.751763 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.751740    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:18:53.879999 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.879875    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a9973e-c077-4cde-a8bc-667bdbf62fbb-openshift-service-ca\") pod \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") "
Jul 03 21:18:53.879999 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.879967    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fq2b\" (UniqueName: \"kubernetes.io/projected/17a9973e-c077-4cde-a8bc-667bdbf62fbb-kube-api-access-7fq2b\") pod \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") "
Jul 03 21:18:53.880237 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.880021    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a9973e-c077-4cde-a8bc-667bdbf62fbb-maas-api-tls\") pod \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\" (UID: \"17a9973e-c077-4cde-a8bc-667bdbf62fbb\") "
Jul 03 21:18:53.880290 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.880250    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a9973e-c077-4cde-a8bc-667bdbf62fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17a9973e-c077-4cde-a8bc-667bdbf62fbb" (UID: "17a9973e-c077-4cde-a8bc-667bdbf62fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 21:18:53.882263 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.882233    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a9973e-c077-4cde-a8bc-667bdbf62fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17a9973e-c077-4cde-a8bc-667bdbf62fbb" (UID: "17a9973e-c077-4cde-a8bc-667bdbf62fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 03 21:18:53.882406 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.882390    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a9973e-c077-4cde-a8bc-667bdbf62fbb-kube-api-access-7fq2b" (OuterVolumeSpecName: "kube-api-access-7fq2b") pod "17a9973e-c077-4cde-a8bc-667bdbf62fbb" (UID: "17a9973e-c077-4cde-a8bc-667bdbf62fbb"). InnerVolumeSpecName "kube-api-access-7fq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:18:53.981139 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.981107    2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a9973e-c077-4cde-a8bc-667bdbf62fbb-openshift-service-ca\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:53.981139 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.981134    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fq2b\" (UniqueName: \"kubernetes.io/projected/17a9973e-c077-4cde-a8bc-667bdbf62fbb-kube-api-access-7fq2b\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:53.981329 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:53.981147    2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17a9973e-c077-4cde-a8bc-667bdbf62fbb-maas-api-tls\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:18:54.035749 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.035720    2553 generic.go:356] "Generic (PLEG): container finished" podID="17a9973e-c077-4cde-a8bc-667bdbf62fbb" containerID="5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298" exitCode=0
Jul 03 21:18:54.035924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.035773    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"
Jul 03 21:18:54.035924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.035812    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5" event={"ID":"17a9973e-c077-4cde-a8bc-667bdbf62fbb","Type":"ContainerDied","Data":"5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298"}
Jul 03 21:18:54.035924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.035872    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5" event={"ID":"17a9973e-c077-4cde-a8bc-667bdbf62fbb","Type":"ContainerDied","Data":"4324d3ead1bdbd9e44038161ba0e4027f51b54db0c9ff46ca83df68a9e85f91a"}
Jul 03 21:18:54.035924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.035913    2553 scope.go:117] "RemoveContainer" containerID="5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298"
Jul 03 21:18:54.044574 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.044556    2553 scope.go:117] "RemoveContainer" containerID="5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298"
Jul 03 21:18:54.044843 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:18:54.044822    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298\": container with ID starting with 5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298 not found: ID does not exist" containerID="5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298"
Jul 03 21:18:54.044924 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.044853    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298"} err="failed to get container status \"5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298\": rpc error: code = NotFound desc = could not find container \"5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298\": container with ID starting with 5e0a91f7de951fa5ca5911f3f777ba686bc4cca7a0fcdc83a4a894a304894298 not found: ID does not exist"
Jul 03 21:18:54.057454 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.057428    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"]
Jul 03 21:18:54.063729 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.063707    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-26641f-7b787d575d-h7wg5"]
Jul 03 21:18:54.870703 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:54.870670    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a9973e-c077-4cde-a8bc-667bdbf62fbb" path="/var/lib/kubelet/pods/17a9973e-c077-4cde-a8bc-667bdbf62fbb/volumes"
Jul 03 21:18:59.079208 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.079173    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:59.131200 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.131154    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:59.312766 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.312735    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"]
Jul 03 21:18:59.313024 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.312998    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy" containerID="cri-o://203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51" gracePeriod=30
Jul 03 21:18:59.333119 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.333029    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:59.544661 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.544627    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:18:59.987166 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:18:59.987135    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f754c655b-k6lnj_bc970611-5804-4465-b9ed-3747c018a172/authorino/0.log"
Jul 03 21:19:04.497958 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.497927    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v6cj5_859207ed-804c-4ab1-bf88-2b745e381487/manager/0.log"
Jul 03 21:19:04.567065 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.567042    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:19:04.619138 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.619066    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bxf2b_8fc5fe7c-f029-48eb-89f7-280833e21ba2/manager/0.log"
Jul 03 21:19:04.682337 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682307    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-data\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682487 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682346    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-podinfo\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682487 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682378    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-credential-socket\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682487 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682397    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-token\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682487 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682424    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-socket\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682490    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istiod-ca-cert\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682538    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-certs\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682560    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-envoy\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682621    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfk2f\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-kube-api-access-lfk2f\") pod \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\" (UID: \"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7\") "
Jul 03 21:19:04.682733 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682692    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:19:04.683001 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682772    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:19:04.683001 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682801    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:19:04.683001 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682871    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 03 21:19:04.683001 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.682991    2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istiod-ca-cert\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.683190 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.683012    2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-certs\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.683190 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.683026    2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-credential-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.683190 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.683042    2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-workload-socket\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.683407 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.683375    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:19:04.684714 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.684689    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-kube-api-access-lfk2f" (OuterVolumeSpecName: "kube-api-access-lfk2f") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "kube-api-access-lfk2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:19:04.684799 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.684694    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 03 21:19:04.684923 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.684889    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 03 21:19:04.685050 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.685027    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" (UID: "0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 03 21:19:04.784097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.784067    2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-envoy\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.784097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.784092    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfk2f\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-kube-api-access-lfk2f\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.784097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.784102    2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-data\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.784299 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.784110    2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-podinfo\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:04.784299 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:04.784118    2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7-istio-token\") on node \"ip-10-0-129-236.ec2.internal\" DevicePath \"\""
Jul 03 21:19:05.089842 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.089809    2553 generic.go:356] "Generic (PLEG): container finished" podID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerID="203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51" exitCode=0
Jul 03 21:19:05.090045 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.089857    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"
Jul 03 21:19:05.090045 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.089914    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" event={"ID":"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7","Type":"ContainerDied","Data":"203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51"}
Jul 03 21:19:05.090045 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.089971    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh" event={"ID":"0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7","Type":"ContainerDied","Data":"a6f85b7f6f3ac7abeb49af3becf1029147afc73fe3f8d168ebf332e82c2640d3"}
Jul 03 21:19:05.090045 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.090001    2553 scope.go:117] "RemoveContainer" containerID="203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51"
Jul 03 21:19:05.098704 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.098686    2553 scope.go:117] "RemoveContainer" containerID="9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"
Jul 03 21:19:05.106853 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.106823    2553 scope.go:117] "RemoveContainer" containerID="203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51"
Jul 03 21:19:05.107171 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:19:05.107153    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51\": container with ID starting with 203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51 not found: ID does not exist" containerID="203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51"
Jul 03 21:19:05.107251 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.107180    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51"} err="failed to get container status \"203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51\": rpc error: code = NotFound desc = could not find container \"203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51\": container with ID starting with 203162acdd13c0b19551b1f45fe91c03b8f9a83d0a605124d4f6db530a282e51 not found: ID does not exist"
Jul 03 21:19:05.107251 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.107197    2553 scope.go:117] "RemoveContainer" containerID="9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"
Jul 03 21:19:05.107475 ip-10-0-129-236 kubenswrapper[2553]: E0703 21:19:05.107459    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5\": container with ID starting with 9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5 not found: ID does not exist" containerID="9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"
Jul 03 21:19:05.107526 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.107479    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5"} err="failed to get container status \"9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5\": rpc error: code = NotFound desc = could not find container \"9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5\": container with ID starting with 9b1be2f7d2061ac5ec939ff680cc5b80b7c111be357371c6a78a3317a270baa5 not found: ID does not exist"
Jul 03 21:19:05.108500 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.108482    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"]
Jul 03 21:19:05.111599 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.111577    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-26641f-openshift-default-6dd68cd7c5-vncjh"]
Jul 03 21:19:05.198459 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.198394    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-92l6h_7011f91d-9e43-4583-9d1c-904fef74566e/server/0.log"
Jul 03 21:19:05.324316 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.324290    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8fsbz_ce70dfc8-14cb-4833-912e-7c0cadd96d09/manager/0.log"
Jul 03 21:19:05.694771 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.694740    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5448f4d7cf-vwdqt_4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155/copy-manifests/0.log"
Jul 03 21:19:05.713112 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:05.713086    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5448f4d7cf-vwdqt_4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155/manager/0.log"
Jul 03 21:19:06.870660 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:06.870628    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" path="/var/lib/kubelet/pods/0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7/volumes"
Jul 03 21:19:07.074974 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:07.074941    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f754c655b-k6lnj_bc970611-5804-4465-b9ed-3747c018a172/authorino/0.log"
Jul 03 21:19:07.209910 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:07.209792    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-rs5vq_41637a53-f3b5-4e15-9983-937e9ed32138/manager/0.log"
Jul 03 21:19:07.319365 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:07.319340    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kzv4h_0c44cc94-1784-4508-b007-3a12a7887e5a/manager/0.log"
Jul 03 21:19:07.544593 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:07.544564    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-x5fk2_31977d5a-0150-4f3c-a532-9340ff04ac16/registry-server/0.log"
Jul 03 21:19:07.670691 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:07.670658    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rz4fr_d0db8825-f5ee-400a-bcff-5930dd67b09b/manager/0.log"
Jul 03 21:19:07.898391 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:07.898310    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jp5rq_c861b0e8-54fe-4ad3-828f-66ef4a50a9dd/manager/0.log"
Jul 03 21:19:08.252751 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:08.252664    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678l44k5_b5cfadb5-2d8b-410a-905f-518862b2d5b3/istio-proxy/0.log"
Jul 03 21:19:08.588229 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:08.588200    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff55469-vpmvc_a2f6782e-69ba-436a-b291-948e6aa42a1c/kube-auth-proxy/0.log"
Jul 03 21:19:08.712696 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:08.712661    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/1.log"
Jul 03 21:19:09.393730 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:09.393702    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jd97h_e6ed86e7-226a-4386-97eb-c64e7f4870e1/main/0.log"
Jul 03 21:19:09.840548 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:09.840520    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hqk6t_7b539ac1-8323-42e1-a854-403e03cbb65b/main/0.log"
Jul 03 21:19:13.994474 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.994437    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nbmfn/must-gather-8xvtw"]
Jul 03 21:19:13.994958 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.994940    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a9973e-c077-4cde-a8bc-667bdbf62fbb" containerName="maas-api"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.994962    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a9973e-c077-4cde-a8bc-667bdbf62fbb" containerName="maas-api"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.994974    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69c9ea3-86a6-48ea-8133-f15467c9d642" containerName="main"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.994981    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69c9ea3-86a6-48ea-8133-f15467c9d642" containerName="main"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.994993    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ee111e-d404-4b4e-ab85-c8971b4b06f3" containerName="main"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995002    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ee111e-d404-4b4e-ab85-c8971b4b06f3" containerName="main"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995016    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy"
Jul 03 21:19:13.995038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995024    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995046    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995054    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995145    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995156    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69c9ea3-86a6-48ea-8133-f15467c9d642" containerName="main"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995165    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ee111e-d404-4b4e-ab85-c8971b4b06f3" containerName="main"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995175    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a9973e-c077-4cde-a8bc-667bdbf62fbb" containerName="maas-api"
Jul 03 21:19:13.995375 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:13.995183    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a79820b-aa2b-4f0d-ab5b-9b6ce2fdd4f7" containerName="istio-proxy"
Jul 03 21:19:14.000084 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.000063    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.002758 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.002727    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nbmfn\"/\"default-dockercfg-sb5rk\""
Jul 03 21:19:14.002931 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.002757    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nbmfn\"/\"openshift-service-ca.crt\""
Jul 03 21:19:14.004242 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.004222    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nbmfn\"/\"kube-root-ca.crt\""
Jul 03 21:19:14.006307 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.006280    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:19:14.007237 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.007212    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbmfn/must-gather-8xvtw"]
Jul 03 21:19:14.057115 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.057082    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log" limit=1024
Jul 03 21:19:14.170750 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.170723    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27dz\" (UniqueName: \"kubernetes.io/projected/81ff6639-c56b-406e-8b0c-d55908278a68-kube-api-access-t27dz\") pod \"must-gather-8xvtw\" (UID: \"81ff6639-c56b-406e-8b0c-d55908278a68\") " pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.170940 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.170768    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81ff6639-c56b-406e-8b0c-d55908278a68-must-gather-output\") pod \"must-gather-8xvtw\" (UID: \"81ff6639-c56b-406e-8b0c-d55908278a68\") " pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.271970 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.271859    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27dz\" (UniqueName: \"kubernetes.io/projected/81ff6639-c56b-406e-8b0c-d55908278a68-kube-api-access-t27dz\") pod \"must-gather-8xvtw\" (UID: \"81ff6639-c56b-406e-8b0c-d55908278a68\") " pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.271970 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.271950    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81ff6639-c56b-406e-8b0c-d55908278a68-must-gather-output\") pod \"must-gather-8xvtw\" (UID: \"81ff6639-c56b-406e-8b0c-d55908278a68\") " pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.272281 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.272261    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/81ff6639-c56b-406e-8b0c-d55908278a68-must-gather-output\") pod \"must-gather-8xvtw\" (UID: \"81ff6639-c56b-406e-8b0c-d55908278a68\") " pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.279922 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.279886    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27dz\" (UniqueName: \"kubernetes.io/projected/81ff6639-c56b-406e-8b0c-d55908278a68-kube-api-access-t27dz\") pod \"must-gather-8xvtw\" (UID: \"81ff6639-c56b-406e-8b0c-d55908278a68\") " pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.309948 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.309915    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbmfn/must-gather-8xvtw"
Jul 03 21:19:14.439292 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.439267    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbmfn/must-gather-8xvtw"]
Jul 03 21:19:14.441355 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:19:14.441325    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81ff6639_c56b_406e_8b0c_d55908278a68.slice/crio-f56b283bd66ccc8c8fd35c709919b8768c0334d15c45ac1058d091aad688e72b WatchSource:0}: Error finding container f56b283bd66ccc8c8fd35c709919b8768c0334d15c45ac1058d091aad688e72b: Status 404 returned error can't find the container with id f56b283bd66ccc8c8fd35c709919b8768c0334d15c45ac1058d091aad688e72b
Jul 03 21:19:14.443219 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:14.443203    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 03 21:19:15.143775 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:15.143739    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbmfn/must-gather-8xvtw" event={"ID":"81ff6639-c56b-406e-8b0c-d55908278a68","Type":"ContainerStarted","Data":"f56b283bd66ccc8c8fd35c709919b8768c0334d15c45ac1058d091aad688e72b"}
Jul 03 21:19:16.150310 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:16.150271    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbmfn/must-gather-8xvtw" event={"ID":"81ff6639-c56b-406e-8b0c-d55908278a68","Type":"ContainerStarted","Data":"6dd5f1cd1fd3b2646f121c1fb4c33b23dec04d3a42189b8d112d100cebd6cdf0"}
Jul 03 21:19:16.150780 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:16.150323    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbmfn/must-gather-8xvtw" event={"ID":"81ff6639-c56b-406e-8b0c-d55908278a68","Type":"ContainerStarted","Data":"13613bb5acd1407b28f90fb40a8e3843bb9afaa8a73b7752ad87b99e0f082af0"}
Jul 03 21:19:16.166226 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:16.166180    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nbmfn/must-gather-8xvtw" podStartSLOduration=2.207350915 podStartE2EDuration="3.16616441s" totalImagesPullingTime="958.813495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 21:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:19:16.16542431 +0000 UTC m=+2528.246165719" watchObservedRunningTime="2026-07-03 21:19:16.16616441 +0000 UTC m=+2528.246905818"
Jul 03 21:19:16.920532 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:16.920499    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xwkdt_55d54b63-00e7-4ef2-b36b-0cc61d1a434c/global-pull-secret-syncer/0.log"
Jul 03 21:19:16.965423 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:16.965392    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4bgfb_a3c68222-8adb-44f7-a47a-029f466be0a3/konnectivity-agent/0.log"
Jul 03 21:19:17.060141 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:17.060109    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-236.ec2.internal_4d17de08da950bb8f59f5515256e0343/haproxy/0.log"
Jul 03 21:19:22.348914 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:22.348015    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f754c655b-k6lnj_bc970611-5804-4465-b9ed-3747c018a172/authorino/0.log"
Jul 03 21:19:22.389715 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:22.389682    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-rs5vq_41637a53-f3b5-4e15-9983-937e9ed32138/manager/0.log"
Jul 03 21:19:22.422732 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:22.422698    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kzv4h_0c44cc94-1784-4508-b007-3a12a7887e5a/manager/0.log"
Jul 03 21:19:22.496286 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:22.496252    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-x5fk2_31977d5a-0150-4f3c-a532-9340ff04ac16/registry-server/0.log"
Jul 03 21:19:22.649235 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:22.649084    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rz4fr_d0db8825-f5ee-400a-bcff-5930dd67b09b/manager/0.log"
Jul 03 21:19:22.804539 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:22.804189    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jp5rq_c861b0e8-54fe-4ad3-828f-66ef4a50a9dd/manager/0.log"
Jul 03 21:19:24.721038 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:24.721002    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6nkjq_e29d6023-125a-455c-8443-af0ccc5d6453/kube-state-metrics/0.log"
Jul 03 21:19:24.740969 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:24.740933    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6nkjq_e29d6023-125a-455c-8443-af0ccc5d6453/kube-rbac-proxy-main/0.log"
Jul 03 21:19:24.756043 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:24.756010    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6nkjq_e29d6023-125a-455c-8443-af0ccc5d6453/kube-rbac-proxy-self/0.log"
Jul 03 21:19:24.980705 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:24.980603    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qdsm_dc2eec07-d321-4093-a18f-a2198810cc59/node-exporter/0.log"
Jul 03 21:19:24.996114 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:24.996072    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qdsm_dc2eec07-d321-4093-a18f-a2198810cc59/kube-rbac-proxy/0.log"
Jul 03 21:19:25.014818 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:25.014775    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qdsm_dc2eec07-d321-4093-a18f-a2198810cc59/init-textfile/0.log"
Jul 03 21:19:27.333532 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.333497    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"]
Jul 03 21:19:27.341265 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.341228    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.344352 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.344322    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"]
Jul 03 21:19:27.416527 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.416496    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-podres\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.416726 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.416558    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-sys\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.416726 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.416598    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85j8\" (UniqueName: \"kubernetes.io/projected/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-kube-api-access-s85j8\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.416726 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.416650    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-lib-modules\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.416845 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.416754    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-proc\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517487 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517450    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85j8\" (UniqueName: \"kubernetes.io/projected/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-kube-api-access-s85j8\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-lib-modules\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517565    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-proc\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517639    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-lib-modules\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517644    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-podres\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517695    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-proc\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517743    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-podres\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517744    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-sys\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.517833 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.517795    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-sys\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.526854 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.525950    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85j8\" (UniqueName: \"kubernetes.io/projected/d1dca74a-bfb6-4fd7-82b8-340e9d80fc72-kube-api-access-s85j8\") pod \"perf-node-gather-daemonset-2g4dj\" (UID: \"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72\") " pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.660940 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.660824    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:27.832751 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:27.832723    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"]
Jul 03 21:19:27.834662 ip-10-0-129-236 kubenswrapper[2553]: W0703 21:19:27.834629    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1dca74a_bfb6_4fd7_82b8_340e9d80fc72.slice/crio-4b2770144dc98b476746cfbe33e12fd503d73466221dd1b7961a083ee3a32afd WatchSource:0}: Error finding container 4b2770144dc98b476746cfbe33e12fd503d73466221dd1b7961a083ee3a32afd: Status 404 returned error can't find the container with id 4b2770144dc98b476746cfbe33e12fd503d73466221dd1b7961a083ee3a32afd
Jul 03 21:19:28.247985 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:28.247868    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj" event={"ID":"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72","Type":"ContainerStarted","Data":"635ba5067158a0c938b40b7ed6c0eaee98e3404e093d23d571772f64787ddc42"}
Jul 03 21:19:28.247985 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:28.247960    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj" event={"ID":"d1dca74a-bfb6-4fd7-82b8-340e9d80fc72","Type":"ContainerStarted","Data":"4b2770144dc98b476746cfbe33e12fd503d73466221dd1b7961a083ee3a32afd"}
Jul 03 21:19:28.248174 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:28.247992    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:28.264501 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:28.264454    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj" podStartSLOduration=1.264439097 podStartE2EDuration="1.264439097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 21:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 21:19:28.26253699 +0000 UTC m=+2540.343278399" watchObservedRunningTime="2026-07-03 21:19:28.264439097 +0000 UTC m=+2540.345180546"
Jul 03 21:19:29.186727 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:29.186693    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pz4zv_69053652-4d62-46a7-b20e-7a963f70b45d/dns/0.log"
Jul 03 21:19:29.202181 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:29.202151    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pz4zv_69053652-4d62-46a7-b20e-7a963f70b45d/kube-rbac-proxy/0.log"
Jul 03 21:19:29.290077 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:29.290052    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpjjr_c84e01a7-e24c-4313-9fb4-636dc71bc1b2/dns-node-resolver/0.log"
Jul 03 21:19:29.801370 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:29.801339    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d9d77fc47-7m4sv_e9850e83-282e-40b9-aee8-b26092d7be38/registry/0.log"
Jul 03 21:19:29.859330 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:29.859305    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rcmn4_7e751430-2992-4aeb-b4c3-15660c5f3ffa/node-ca/0.log"
Jul 03 21:19:30.715391 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:30.715355    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678l44k5_b5cfadb5-2d8b-410a-905f-518862b2d5b3/istio-proxy/0.log"
Jul 03 21:19:30.853682 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:30.853642    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff55469-vpmvc_a2f6782e-69ba-436a-b291-948e6aa42a1c/kube-auth-proxy/0.log"
Jul 03 21:19:30.902414 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:30.902385    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/0.log"
Jul 03 21:19:30.962556 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:30.962525    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/1.log"
Jul 03 21:19:31.706755 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:31.706725    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6nz9p_becd33cb-180b-42fe-bccb-161a55b71620/serve-healthcheck-canary/0.log"
Jul 03 21:19:32.313693 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:32.313663    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vpgmp_b39e2730-3085-4abf-a0e4-4f0c308a3deb/kube-rbac-proxy/0.log"
Jul 03 21:19:32.328816 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:32.328789    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vpgmp_b39e2730-3085-4abf-a0e4-4f0c308a3deb/exporter/0.log"
Jul 03 21:19:32.346240 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:32.346213    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vpgmp_b39e2730-3085-4abf-a0e4-4f0c308a3deb/extractor/0.log"
Jul 03 21:19:34.266448 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:34.266414    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nbmfn/perf-node-gather-daemonset-2g4dj"
Jul 03 21:19:34.371031 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:34.370999    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-v6cj5_859207ed-804c-4ab1-bf88-2b745e381487/manager/0.log"
Jul 03 21:19:34.626615 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:34.626581    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bxf2b_8fc5fe7c-f029-48eb-89f7-280833e21ba2/manager/0.log"
Jul 03 21:19:34.802565 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:34.802539    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-92l6h_7011f91d-9e43-4583-9d1c-904fef74566e/server/0.log"
Jul 03 21:19:34.910557 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:34.910463    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8fsbz_ce70dfc8-14cb-4833-912e-7c0cadd96d09/manager/0.log"
Jul 03 21:19:35.106687 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:35.106657    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5448f4d7cf-vwdqt_4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155/manager/0.log"
Jul 03 21:19:35.121041 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:35.121017    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5448f4d7cf-vwdqt_4ed4cc2b-e67f-42f7-9ef3-c3f23caf0155/copy-manifests/0.log"
Jul 03 21:19:36.601623 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:36.601533    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kc5hl_ee77cc6c-d5c9-47a2-9527-3313cd65e689/openshift-lws-operator/0.log"
Jul 03 21:19:37.670306 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:37.670222    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678l44k5_b5cfadb5-2d8b-410a-905f-518862b2d5b3/istio-proxy/0.log"
Jul 03 21:19:37.787162 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:37.787131    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff55469-vpmvc_a2f6782e-69ba-436a-b291-948e6aa42a1c/kube-auth-proxy/0.log"
Jul 03 21:19:37.802312 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:37.802281    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/0.log"
Jul 03 21:19:37.855847 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:37.855807    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/1.log"
Jul 03 21:19:39.154485 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:39.154451    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678l44k5_b5cfadb5-2d8b-410a-905f-518862b2d5b3/istio-proxy/0.log"
Jul 03 21:19:39.282715 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:39.282688    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dff55469-vpmvc_a2f6782e-69ba-436a-b291-948e6aa42a1c/kube-auth-proxy/0.log"
Jul 03 21:19:39.301856 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:39.301812    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/0.log"
Jul 03 21:19:39.363913 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:39.363868    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-6jcdd_8c63b7a3-5b6c-4e4f-9551-1af0d38dd772/istio-proxy/1.log"
Jul 03 21:19:42.878294 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.878260    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9p4vf_f11ddbe1-9a74-4630-ba56-313c42396515/kube-multus/0.log"
Jul 03 21:19:42.900365 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.900336    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/kube-multus-additional-cni-plugins/0.log"
Jul 03 21:19:42.915713 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.915690    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/egress-router-binary-copy/0.log"
Jul 03 21:19:42.931256 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.931234    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/cni-plugins/0.log"
Jul 03 21:19:42.945907 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.945867    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/bond-cni-plugin/0.log"
Jul 03 21:19:42.980097 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.980074    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/routeoverride-cni/0.log"
Jul 03 21:19:42.995795 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:42.995771    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/whereabouts-cni-bincopy/0.log"
Jul 03 21:19:43.011148 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:43.011119    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jsnzb_2fa04aaa-635c-4e0b-ba6c-f871684417cb/whereabouts-cni/0.log"
Jul 03 21:19:43.401512 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:43.401481    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ts9j8_23f92759-61cf-4a1d-8b4a-c5304c4c0c59/network-metrics-daemon/0.log"
Jul 03 21:19:43.413391 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:43.413364    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ts9j8_23f92759-61cf-4a1d-8b4a-c5304c4c0c59/kube-rbac-proxy/0.log"
Jul 03 21:19:44.372333 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.372253    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/ovn-controller/0.log"
Jul 03 21:19:44.385568 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.385538    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/ovn-acl-logging/0.log"
Jul 03 21:19:44.411950 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.411919    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/kube-rbac-proxy-node/0.log"
Jul 03 21:19:44.434297 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.434269    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 03 21:19:44.449094 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.449065    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/northd/0.log"
Jul 03 21:19:44.464198 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.464172    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/nbdb/0.log"
Jul 03 21:19:44.482247 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.482208    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/sbdb/0.log"
Jul 03 21:19:44.570244 ip-10-0-129-236 kubenswrapper[2553]: I0703 21:19:44.570217    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l4mn2_5871fbca-41b9-4cfc-bd4a-4d9d0234b6e2/ovnkube-controller/0.log"