Sep 11 17:52:14.018656 ip-10-0-141-255 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 17:52:14.018669 ip-10-0-141-255 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 17:52:14.018679 ip-10-0-141-255 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 17:52:14.019010 ip-10-0-141-255 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 17:52:24.060903 ip-10-0-141-255 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 17:52:24.060920 ip-10-0-141-255 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1789061b6952470583c6f2bcfc90c7ac -- Sep 11 17:54:15.845706 ip-10-0-141-255 systemd[1]: Starting Kubernetes Kubelet... Sep 11 17:54:16.251012 ip-10-0-141-255 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 17:54:16.251012 ip-10-0-141-255 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 17:54:16.251012 ip-10-0-141-255 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 17:54:16.251012 ip-10-0-141-255 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 17:54:16.251012 ip-10-0-141-255 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 17:54:16.251881 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.251787 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 17:54:16.256875 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256855 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 17:54:16.256875 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256872 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 17:54:16.256875 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256876 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 17:54:16.256875 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256882 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256886 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256890 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256894 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256898 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256902 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256905 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256909 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256912 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256916 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256927 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256932 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256936 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256940 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256943 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256947 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256951 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256955 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256958 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256962 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 17:54:16.257123 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256966 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256970 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256974 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256977 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256981 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256985 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256989 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256992 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.256997 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257004 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257010 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257015 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257021 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257026 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257031 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257041 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257046 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257051 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257055 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257059 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 17:54:16.257974 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257063 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257068 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257072 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257076 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257080 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257085 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257089 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257094 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257098 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257102 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257106 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257113 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257119 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257123 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257127 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257131 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257135 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257139 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257144 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257148 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 17:54:16.258883 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257152 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257156 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257160 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257165 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257169 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257173 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257178 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257183 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257195 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257200 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257204 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257208 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257213 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257217 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257222 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257226 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257230 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257236 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257240 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 17:54:16.259737 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257244 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257251 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257256 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257261 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257266 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257271 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257275 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257280 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257284 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257288 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257293 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257296 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257300 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257304 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257308 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257313 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.257317 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258603 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258613 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 17:54:16.260573 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258619 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258624 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258632 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258639 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258645 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258649 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258654 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258658 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258662 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258666 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258670 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258674 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258678 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258683 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258687 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258691 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258695 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258699 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258703 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258708 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 17:54:16.261385 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258712 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258716 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258721 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258725 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258729 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258734 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258738 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258743 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258748 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258752 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258757 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258762 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258767 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258771 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258775 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258779 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258784 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258789 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258793 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258797 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 17:54:16.262193 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258801 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258806 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258810 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258814 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258818 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258822 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258826 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258831 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258835 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258839 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258843 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258847 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258851 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258855 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258864 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258870 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258875 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258879 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258884 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 17:54:16.262785 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258888 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258892 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258897 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258902 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258907 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258911 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258915 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258919 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258924 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258928 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258934 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258939 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258943 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258947 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258951 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258955 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258959 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258963 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258967 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 17:54:16.263280 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258972 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258976 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258980 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258985 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258990 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.258996 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259001 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259005 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259009 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259013 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259018 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259022 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259027 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259031 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259035 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259039 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259043 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259047 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259051 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259707 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 17:54:16.263945 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259717 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259723 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259728 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259733 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259738 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259742 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259745 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259749 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259754 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259758 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259762 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259766 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259770 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259776 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259780 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259784 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259788 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259795 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259801 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259806 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 17:54:16.264502 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259810 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259815 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259819 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259824 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259829 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259834 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259838 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259843 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259847 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259852 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259857 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259861 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259865 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259870 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259874 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259878 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259884 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259888 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259892 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259896 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 17:54:16.265013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259904 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259908 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259913 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259917 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259921 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259927 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259932 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259935 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259940 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259944 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259948 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259952 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259956 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259961 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259965 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259970 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259974 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259978 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259982 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 17:54:16.265650 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259986 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259993 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.259998 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260002 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260006 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260010 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260014 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260018 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260023 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260027 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260032 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260036 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260040 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260046 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260050 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260054 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260058 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260063 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260068 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260072 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 17:54:16.266128 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260076 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260082 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260087 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260092 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260100 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260104 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260108 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260112 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260117 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260121 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260125 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260129 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260133 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260137 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260141 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260145 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260150 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260154 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.260158 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260272 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 17:54:16.266651 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260322 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260333 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260339 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260347 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260352 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260361 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260374 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260379 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260384 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260389 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260394 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260399 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260404 2579 flags.go:64] FLAG: --cgroup-root="" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260409 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260437 2579 flags.go:64] FLAG: --client-ca-file="" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260442 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260448 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260455 2579 flags.go:64] FLAG: --cluster-domain="" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260459 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260464 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260469 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260475 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260482 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 17:54:16.267136 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260486 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260491 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260497 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260501 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260506 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260511 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260516 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260521 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260527 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260533 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260538 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260542 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260547 2579 flags.go:64] FLAG: --enable-server="true" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260553 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260563 2579 flags.go:64] FLAG: --event-burst="100" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260568 2579 flags.go:64] FLAG: --event-qps="50" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260572 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260577 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260581 2579 flags.go:64] FLAG: --eviction-hard="" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260587 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260592 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260596 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260601 2579 flags.go:64] FLAG: --eviction-soft="" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260606 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260611 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 17:54:16.267706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260616 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260621 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260626 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260630 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260634 2579 flags.go:64] FLAG: --feature-gates="" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260641 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260645 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260650 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260655 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260660 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260665 2579 flags.go:64] FLAG: --help="false" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260669 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260674 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260679 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260684 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260689 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260695 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260699 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260704 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260708 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260713 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260719 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260724 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260728 2579 flags.go:64] FLAG: --kube-reserved="" Sep 11 17:54:16.268459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260733 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260738 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260743 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260747 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260752 2579 flags.go:64] FLAG: --lock-file="" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260756 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260761 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260766 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260775 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260779 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260784 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260789 2579 flags.go:64] FLAG: --logging-format="text" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260794 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260799 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260804 2579 flags.go:64] FLAG: --manifest-url="" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260808 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260815 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260819 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260826 2579 flags.go:64] FLAG: --max-pods="110" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260830 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260835 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260840 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260844 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260849 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260853 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 17:54:16.269645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260858 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260870 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260874 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260879 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260884 2579 flags.go:64] FLAG: --pod-cidr="" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260890 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260899 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260904 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260909 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260913 2579 flags.go:64] FLAG: --port="10250" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260918 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260923 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05c1db2dc19121f78" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260928 2579 flags.go:64] FLAG: --qos-reserved="" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260934 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260939 2579 flags.go:64] FLAG: --register-node="true" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260944 2579 flags.go:64] FLAG: --register-with-taints="" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260950 2579 flags.go:64] FLAG: --registry-burst="10" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260954 2579 flags.go:64] FLAG: --registry-qps="5" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260959 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260963 2579 flags.go:64] FLAG: --reserved-memory="" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260969 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260974 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260979 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260983 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260988 2579 flags.go:64] FLAG: --runonce="false" Sep 11 17:54:16.270745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260992 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.260997 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261002 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261007 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261011 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261016 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261020 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261025 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261030 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261034 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261039 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261043 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261048 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261054 2579 flags.go:64] FLAG: --system-cgroups="" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261078 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261089 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261095 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261101 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261108 2579 flags.go:64] FLAG: --tls-min-version="" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261113 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261118 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261122 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261127 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261132 2579 flags.go:64] FLAG: --v="2" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261138 2579 flags.go:64] FLAG: --version="false" Sep 11 17:54:16.271607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261145 2579 flags.go:64] FLAG: --vmodule="" Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261151 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.261157 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261350 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261357 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261361 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261366 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261371 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261375 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261379 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261383 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261388 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261392 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261396 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261400 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261404 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261408 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261412 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261437 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261441 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 17:54:16.272313 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261445 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261450 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261457 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261463 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261468 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261473 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261477 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261481 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261488 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261494 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261501 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261505 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261509 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261513 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261518 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261521 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261526 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261530 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261534 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 17:54:16.273195 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261538 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261542 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261546 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261550 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261555 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261559 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261562 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261567 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261571 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261577 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261581 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261585 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261589 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261593 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261598 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261603 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261607 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261611 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261615 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261619 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 17:54:16.274013 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261624 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261628 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261632 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261638 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261642 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261647 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261652 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261656 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261661 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261664 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261669 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261673 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261677 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261681 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261685 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261689 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261693 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261698 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261702 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261706 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 17:54:16.274757 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261711 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261717 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261721 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261726 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261730 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261734 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261738 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261745 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261751 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261756 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261761 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261765 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261769 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261774 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261778 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261785 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261789 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261793 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261797 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 17:54:16.275317 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261802 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261806 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261810 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.261814 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.262383 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.271235 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.271256 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.272094 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.272107 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272205 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272212 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272218 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272222 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272227 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272232 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 17:54:16.275809 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272236 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272241 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272245 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272250 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272255 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272259 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272263 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272267 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272271 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272276 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272280 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272284 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272288 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272292 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272300 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272307 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272311 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272316 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272320 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 17:54:16.276181 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272325 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272329 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272334 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272338 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272351 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272355 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272360 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272364 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272368 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272372 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272377 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272381 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272385 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272389 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272394 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272398 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272402 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272406 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272410 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272438 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 17:54:16.276665 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272443 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272447 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272452 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272457 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272463 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272469 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272473 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272478 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272483 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272486 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272490 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272494 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272498 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272502 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272507 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272511 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272514 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272527 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272533 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 17:54:16.277153 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272537 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272541 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272545 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272549 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272553 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272558 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272562 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272566 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272570 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272576 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272582 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272586 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272591 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272595 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272599 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272603 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272608 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272612 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272616 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272621 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 17:54:16.277651 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272625 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272629 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272633 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272637 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272641 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272645 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272649 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272653 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272657 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272661 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272665 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272677 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272681 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272685 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272689 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 17:54:16.278126 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.272697 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272939 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272950 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272956 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272961 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272965 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272970 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272974 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272978 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272982 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272987 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272991 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272995 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.272999 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273003 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273008 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273012 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273016 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273020 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 17:54:16.278577 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273024 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273029 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273033 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273037 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273041 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273045 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273050 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273054 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273058 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273062 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273076 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273084 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273090 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273094 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273099 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273103 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273107 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273111 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273115 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 17:54:16.279034 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273120 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273124 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273128 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273132 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273136 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273141 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273145 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273149 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273153 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273157 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273161 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273165 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273169 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273174 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273178 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273182 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273186 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273190 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273194 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273198 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 17:54:16.279513 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273202 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273206 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273210 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273213 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273225 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273230 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273234 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273238 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273242 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273246 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273251 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273255 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273259 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273263 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273267 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273271 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273275 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273280 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273284 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 17:54:16.279985 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273288 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273292 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273296 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273300 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273304 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273308 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273312 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273316 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273320 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273324 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273328 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273332 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273344 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273348 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273353 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273364 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273370 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273374 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273386 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273390 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 17:54:16.280485 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273394 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273399 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:16.273402 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.273410 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.275194 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.277191 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.278179 2579 server.go:1019] "Starting client certificate rotation" Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.278274 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.278313 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 17:54:16.280973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.280653 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 17:54:16.282610 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.282592 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 17:54:16.292630 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.292608 2579 log.go:25] "Validated CRI v1 runtime API" Sep 11 17:54:16.297719 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.297699 2579 log.go:25] "Validated CRI v1 image API" Sep 11 17:54:16.298552 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.298537 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 17:54:16.301697 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.301675 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b14173d3-2c34-4fa2-955a-62356e6d24d7:/dev/nvme0n1p3 fe8f7731-0b7c-4c30-9712-0ecf591d64aa:/dev/nvme0n1p4] Sep 11 17:54:16.301764 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.301695 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 17:54:16.307620 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.307513 2579 manager.go:216] Machine: {Timestamp:2026-09-11 17:54:16.306153225 +0000 UTC m=+0.337214457 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099634 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21d4ac51c6a7a5a013da0a5706fb13 SystemUUID:ec21d4ac-51c6-a7a5-a013-da0a5706fb13 BootID:1789061b-6952-4705-83c6-f2bcfc90c7ac Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:16:15:45:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:16:15:45:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:88:c3:72:ac:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 17:54:16.307620 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.307616 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 17:54:16.307722 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.307698 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 17:54:16.311181 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.311152 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 17:54:16.311332 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.311184 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-255.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 17:54:16.311922 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.311906 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 17:54:16.311957 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.311940 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 17:54:16.311957 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.311952 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 17:54:16.312017 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.311960 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 17:54:16.312557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.312547 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 17:54:16.313343 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.313327 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 17:54:16.316079 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316068 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 17:54:16.316212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316203 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 17:54:16.316245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316231 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 11 17:54:16.316275 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316245 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 17:54:16.316275 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316258 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 17:54:16.316275 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316266 2579 kubelet.go:398] "Adding apiserver pod source" Sep 11 17:54:16.316275 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316275 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 17:54:16.316700 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316686 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 17:54:16.316776 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.316714 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 17:54:16.318985 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.318969 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 17:54:16.323517 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.323500 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 17:54:16.323585 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.323534 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 17:54:16.325106 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325093 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 17:54:16.325153 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325110 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 17:54:16.325153 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325116 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 17:54:16.325153 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325121 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 17:54:16.325153 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325129 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 17:54:16.325153 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325135 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 17:54:16.325153 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325152 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 17:54:16.325303 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325158 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 17:54:16.325303 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325165 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 17:54:16.325303 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325170 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 17:54:16.325303 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325183 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 17:54:16.325303 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.325188 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 17:54:16.326566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.326549 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 17:54:16.326566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.326566 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 17:54:16.327432 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.327390 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 17:54:16.327514 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.327394 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 17:54:16.330237 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.330221 2579 server.go:1268] "Started kubelet" Sep 11 17:54:16.330322 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.330268 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 17:54:16.330439 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.330374 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 17:54:16.330491 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.330454 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 17:54:16.331141 ip-10-0-141-255 systemd[1]: Started Kubernetes Kubelet. Sep 11 17:54:16.331707 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.331691 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 17:54:16.332458 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.332292 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 11 17:54:16.338436 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.338404 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 17:54:16.338515 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.338439 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 17:54:16.338575 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.338552 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.339015 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.338982 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 17:54:16.339015 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339010 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 17:54:16.339144 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339071 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 17:54:16.339224 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339199 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 11 17:54:16.339224 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339214 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 11 17:54:16.339389 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339367 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-255.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 17:54:16.339565 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339520 2579 factory.go:55] Registering systemd factory Sep 11 17:54:16.339565 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339536 2579 factory.go:223] Registration of the systemd container factory successfully Sep 11 17:54:16.339666 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.339645 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 17:54:16.339736 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339720 2579 factory.go:153] Registering CRI-O factory Sep 11 17:54:16.339736 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339739 2579 factory.go:223] Registration of the crio container factory successfully Sep 11 17:54:16.339866 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339803 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 17:54:16.339866 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339821 2579 factory.go:103] Registering Raw factory Sep 11 17:54:16.339866 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.339838 2579 manager.go:1196] Started watching for new ooms in manager Sep 11 17:54:16.340178 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.339322 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-255.ec2.internal.18d455c4f1b71f72 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-255.ec2.internal,UID:ip-10-0-141-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-255.ec2.internal,},FirstTimestamp:2026-09-11 17:54:16.330190706 +0000 UTC m=+0.361251927,LastTimestamp:2026-09-11 17:54:16.330190706 +0000 UTC m=+0.361251927,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-255.ec2.internal,}" Sep 11 17:54:16.340346 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.340335 2579 manager.go:349] Starting recovery of all containers Sep 11 17:54:16.342600 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.342581 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bcr2h" Sep 11 17:54:16.344913 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.344884 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 17:54:16.344998 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.344981 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 17:54:16.350469 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.350453 2579 manager.go:354] Recovery completed Sep 11 17:54:16.351392 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.351362 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bcr2h" Sep 11 17:54:16.357053 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.357041 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.358844 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.358824 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.358952 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.358860 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.358952 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.358871 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.359453 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.359436 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 17:54:16.359453 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.359451 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 17:54:16.359557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.359455 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.359557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.359465 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 17:54:16.360165 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.360147 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.360235 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.360179 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.360235 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.360191 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.360566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.360551 2579 policy_none.go:49] "None policy: Start" Sep 11 17:54:16.360566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.360569 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 17:54:16.360656 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.360580 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 17:54:16.360948 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.360881 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-255.ec2.internal.18d455c4f36c5dba default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-255.ec2.internal,UID:ip-10-0-141-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-255.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-255.ec2.internal,},FirstTimestamp:2026-09-11 17:54:16.358845882 +0000 UTC m=+0.389907102,LastTimestamp:2026-09-11 17:54:16.358845882 +0000 UTC m=+0.389907102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-255.ec2.internal,}" Sep 11 17:54:16.361641 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.361627 2579 policy_none.go:47] "Start" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.404665 2579 manager.go:337] "Starting Device Plugin manager" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.404696 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.404706 2579 server.go:87] "Starting device plugin registration server" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.404928 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.404938 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.405060 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.405158 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.405167 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.405640 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 17:54:16.407398 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.405683 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.436089 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.436056 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 17:54:16.437369 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.437354 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 17:54:16.437473 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.437397 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 17:54:16.437473 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.437434 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 17:54:16.437582 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.437478 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 17:54:16.439404 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.439386 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 17:54:16.505703 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.505617 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.506609 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.506592 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.506724 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.506630 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.506724 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.506646 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.506724 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.506676 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.514688 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.514671 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.514773 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.514694 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-255.ec2.internal\": node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.523824 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.523801 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.537548 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.537527 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal"] Sep 11 17:54:16.537630 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.537618 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.538446 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.538409 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.538549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.538455 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.538549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.538470 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.540801 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.540788 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.540902 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.540887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.540946 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.540927 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.542150 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.542134 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.542226 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.542163 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.542226 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.542173 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.542226 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.542140 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.542226 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.542197 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.542226 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.542207 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.544443 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.544409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.544492 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.544462 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 17:54:16.546157 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.546140 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientMemory" Sep 11 17:54:16.546238 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.546175 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 17:54:16.546238 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.546190 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeHasSufficientPID" Sep 11 17:54:16.567960 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.567937 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-255.ec2.internal\" not found" node="ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.572225 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.572209 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-255.ec2.internal\" not found" node="ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.623997 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.623973 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.639824 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.639802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd652f262855ed205ed8b8367aab7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal\" (UID: \"3dd652f262855ed205ed8b8367aab7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.639892 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.639837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd652f262855ed205ed8b8367aab7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal\" (UID: \"3dd652f262855ed205ed8b8367aab7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.724910 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.724735 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.740077 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.740048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd652f262855ed205ed8b8367aab7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal\" (UID: \"3dd652f262855ed205ed8b8367aab7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.740186 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.740086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b998e155b62d16440a868b586601783-config\") pod \"kube-apiserver-proxy-ip-10-0-141-255.ec2.internal\" (UID: \"8b998e155b62d16440a868b586601783\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.740186 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.740168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dd652f262855ed205ed8b8367aab7c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal\" (UID: \"3dd652f262855ed205ed8b8367aab7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.740277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.740232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd652f262855ed205ed8b8367aab7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal\" (UID: \"3dd652f262855ed205ed8b8367aab7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.740277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.740266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dd652f262855ed205ed8b8367aab7c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal\" (UID: \"3dd652f262855ed205ed8b8367aab7c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.825535 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.825462 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:16.840733 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.840712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b998e155b62d16440a868b586601783-config\") pod \"kube-apiserver-proxy-ip-10-0-141-255.ec2.internal\" (UID: \"8b998e155b62d16440a868b586601783\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.840789 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.840760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b998e155b62d16440a868b586601783-config\") pod \"kube-apiserver-proxy-ip-10-0-141-255.ec2.internal\" (UID: \"8b998e155b62d16440a868b586601783\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.870879 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.870844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.874645 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:16.874628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" Sep 11 17:54:16.926234 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:16.926205 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:17.026736 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:17.026702 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:17.127345 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:17.127277 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:17.227863 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:17.227830 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:17.278384 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.278357 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 17:54:17.278796 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.278516 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 17:54:17.328942 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:17.328912 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:17.354970 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.354936 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:49:17 +0000 UTC" deadline="2028-04-17 05:21:15.14192678 +0000 UTC" Sep 11 17:54:17.354970 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.354966 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14003h26m57.786963928s" Sep 11 17:54:17.429914 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:17.429850 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-255.ec2.internal\" not found" Sep 11 17:54:17.478655 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.478635 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 17:54:17.484394 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:17.484368 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b998e155b62d16440a868b586601783.slice/crio-f6c699f50ab004c5743094576eb887a3bfa74bc3b2a55d12ba9bd72c0dd4a8d6 WatchSource:0}: Error finding container f6c699f50ab004c5743094576eb887a3bfa74bc3b2a55d12ba9bd72c0dd4a8d6: Status 404 returned error can't find the container with id f6c699f50ab004c5743094576eb887a3bfa74bc3b2a55d12ba9bd72c0dd4a8d6 Sep 11 17:54:17.484659 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:17.484640 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd652f262855ed205ed8b8367aab7c5.slice/crio-d01bf7fade7bfb28c65c7dc951ab07f5cdc25b1d0589283a274af835230ea0e3 WatchSource:0}: Error finding container d01bf7fade7bfb28c65c7dc951ab07f5cdc25b1d0589283a274af835230ea0e3: Status 404 returned error can't find the container with id d01bf7fade7bfb28c65c7dc951ab07f5cdc25b1d0589283a274af835230ea0e3 Sep 11 17:54:17.488892 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.488878 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 17:54:17.538813 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.538780 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" Sep 11 17:54:17.559669 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.559647 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 17:54:17.561517 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.561501 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" Sep 11 17:54:17.578887 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.578867 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 17:54:17.823851 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.823769 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 17:54:17.838052 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:17.838027 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 17:54:18.100287 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.100196 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 17:54:18.317706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.317654 2579 apiserver.go:52] "Watching apiserver" Sep 11 17:54:18.328279 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.328241 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 17:54:18.330681 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.330647 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-z99jr","openshift-cluster-node-tuning-operator/tuned-tspql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal","openshift-multus/multus-gffq8","openshift-multus/network-metrics-daemon-sscnw","openshift-ovn-kubernetes/ovnkube-node-j545j","kube-system/konnectivity-agent-6gxrw","kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2","openshift-dns/node-resolver-t5fnc","openshift-image-registry/node-ca-qrnd9","openshift-multus/multus-additional-cni-plugins-rmxkg","openshift-network-diagnostics/network-check-target-t8wml","openshift-network-operator/iptables-alerter-r9fs8"] Sep 11 17:54:18.333309 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.333289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.335796 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.335757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:18.335921 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.335895 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:18.337229 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.337203 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 17:54:18.337368 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.337203 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 17:54:18.338100 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.338081 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfpw5\"" Sep 11 17:54:18.339287 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.339263 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 17:54:18.339388 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.339341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.341720 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.341699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.343769 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.343235 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 17:54:18.343769 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.343434 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rh8gd\"" Sep 11 17:54:18.344066 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.343995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:18.344135 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.344068 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:18.347644 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.346614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.347644 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.346712 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.347644 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.346729 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 17:54:18.347644 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.347135 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.347644 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.347470 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.347923 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.347847 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 17:54:18.347978 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.347964 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-42pz4\"" Sep 11 17:54:18.348029 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.347999 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.349417 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-host-slash\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.349537 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-cni-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349537 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-k8s-cni-cncf-io\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349537 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-kubelet\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349537 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-hostroot\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349537 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-daemon-config\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbblf\" (UniqueName: \"kubernetes.io/projected/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-kube-api-access-kbblf\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-iptables-alerter-script\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-system-cni-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-cnibin\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85d370f9-8788-4fde-af01-dad42dccac2c-agent-certs\") pod \"konnectivity-agent-6gxrw\" (UID: \"85d370f9-8788-4fde-af01-dad42dccac2c\") " pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85d370f9-8788-4fde-af01-dad42dccac2c-konnectivity-ca\") pod \"konnectivity-agent-6gxrw\" (UID: \"85d370f9-8788-4fde-af01-dad42dccac2c\") " pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-netns\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.349804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-conf-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59j72\" (UniqueName: \"kubernetes.io/projected/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-kube-api-access-59j72\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-cni-binary-copy\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-socket-dir-parent\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-cni-bin\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-cni-multus\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-multus-certs\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.349974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-etc-kubernetes\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.350610 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.350283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.353134 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.352547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.353134 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.352614 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:18.353134 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.352959 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.353134 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.353029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8ctk\"" Sep 11 17:54:18.353400 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.353246 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 17:54:18.353461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.353409 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 17:54:18.353750 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.353731 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 17:54:18.353750 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.353744 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 17:54:18.354017 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.353997 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.355159 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.355139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.356408 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.356376 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 17:54:18.356675 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.356652 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.359207 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.358987 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.360480 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.359509 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qf8gv\"" Sep 11 17:54:18.360480 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.359638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.360480 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.359805 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.360480 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.359966 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.360480 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.360304 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48tgb\"" Sep 11 17:54:18.362830 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.362808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.364940 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.364919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.367963 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.367944 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.370635 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.370615 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ww77d\"" Sep 11 17:54:18.370928 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.370908 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-46tn5\"" Sep 11 17:54:18.371095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.371244 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371231 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9xv89\"" Sep 11 17:54:18.371317 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371302 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 17:54:18.371356 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.370615 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 17:54:18.371495 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371479 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 17:54:18.371642 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371622 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 17:54:18.371757 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371740 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 17:54:18.371940 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.371918 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 17:54:18.395384 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.395360 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnlt7" Sep 11 17:54:18.405509 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.405487 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnlt7" Sep 11 17:54:18.440919 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.440588 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 17:54:18.442858 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.442781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" event={"ID":"3dd652f262855ed205ed8b8367aab7c5","Type":"ContainerStarted","Data":"d01bf7fade7bfb28c65c7dc951ab07f5cdc25b1d0589283a274af835230ea0e3"} Sep 11 17:54:18.444482 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.444452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" event={"ID":"8b998e155b62d16440a868b586601783","Type":"ContainerStarted","Data":"f6c699f50ab004c5743094576eb887a3bfa74bc3b2a55d12ba9bd72c0dd4a8d6"} Sep 11 17:54:18.450649 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-kubelet-config\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.450770 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85d370f9-8788-4fde-af01-dad42dccac2c-agent-certs\") pod \"konnectivity-agent-6gxrw\" (UID: \"85d370f9-8788-4fde-af01-dad42dccac2c\") " pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.450770 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-host-slash\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.450770 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-hostroot\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.450770 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.450930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85d370f9-8788-4fde-af01-dad42dccac2c-konnectivity-ca\") pod \"konnectivity-agent-6gxrw\" (UID: \"85d370f9-8788-4fde-af01-dad42dccac2c\") " pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.450930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.450930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-registration-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.450930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-var-lib-kubelet\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.450930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-conf-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.450930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-run-netns\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovnkube-config\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.450983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-socket-dir-parent\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-kubernetes\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-kubelet\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-env-overrides\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovnkube-script-lib\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-sys\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbblf\" (UniqueName: \"kubernetes.io/projected/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-kube-api-access-kbblf\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.451131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-socket-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjck\" (UniqueName: \"kubernetes.io/projected/712bb0df-b242-4217-9c6b-f74dad8d4767-kube-api-access-6tjck\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-cni-netd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-system-cni-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-device-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-ovn\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovn-node-metrics-cert\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3c2881f-9312-4856-9964-5d0c94db3ea8-hosts-file\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-netns\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-run\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93fceeb3-bb73-4ec4-956d-553a85b9f92a-host\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-systemd-units\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-var-lib-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nkl\" (UniqueName: \"kubernetes.io/projected/e3c2881f-9312-4856-9964-5d0c94db3ea8-kube-api-access-j9nkl\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-cni-bin\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-multus-certs\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.451526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-etc-kubernetes\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysctl-conf\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-etc-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-kubelet\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93fceeb3-bb73-4ec4-956d-553a85b9f92a-serviceca\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-cni-bin\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-system-cni-dir\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-cni-binary-copy\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-iptables-alerter-script\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-cnibin\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-sys-fs\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-lib-modules\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-host\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0792af-21ca-43d1-84d3-529bc7965b8c-tmp\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451921 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 17:54:18.452105 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c2881f-9312-4856-9964-5d0c94db3ea8-tmp-dir\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-conf-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.451961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-systemd\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-kubelet\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-system-cni-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-netns\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-node-log\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-multus-certs\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-etc-kubernetes\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctbb\" (UniqueName: \"kubernetes.io/projected/0068c74d-6fc7-4801-807a-2d9204de6130-kube-api-access-5ctbb\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-hostroot\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-socket-dir-parent\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59j72\" (UniqueName: \"kubernetes.io/projected/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-kube-api-access-59j72\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-cnibin\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-cni-bin\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-host-slash\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-cni-binary-copy\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysctl-d\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.452679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtw8h\" (UniqueName: \"kubernetes.io/projected/9c0792af-21ca-43d1-84d3-529bc7965b8c-kube-api-access-wtw8h\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxskl\" (UniqueName: \"kubernetes.io/projected/93fceeb3-bb73-4ec4-956d-553a85b9f92a-kube-api-access-vxskl\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqsd\" (UniqueName: \"kubernetes.io/projected/63bda03e-2063-492a-aba8-2fc3a351bfcf-kube-api-access-bnqsd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-cni-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-k8s-cni-cncf-io\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-daemon-config\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysconfig\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.452969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgclx\" (UniqueName: \"kubernetes.io/projected/3391cf32-789c-4ed8-80b9-65af2d2a6955-kube-api-access-vgclx\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-modprobe-d\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/85d370f9-8788-4fde-af01-dad42dccac2c-konnectivity-ca\") pod \"konnectivity-agent-6gxrw\" (UID: \"85d370f9-8788-4fde-af01-dad42dccac2c\") " pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-dbus\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-iptables-alerter-script\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.453228 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-run-k8s-cni-cncf-io\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-systemd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-cnibin\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-cni-multus\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-cni-dir\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-cni-binary-copy\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-tuned\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-slash\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-host-var-lib-cni-multus\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-log-socket\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.453877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.453797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-multus-daemon-config\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.456832 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.456808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/85d370f9-8788-4fde-af01-dad42dccac2c-agent-certs\") pod \"konnectivity-agent-6gxrw\" (UID: \"85d370f9-8788-4fde-af01-dad42dccac2c\") " pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.460936 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.460915 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:18.460936 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.460938 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:18.460936 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.460953 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:18.461157 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.461019 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:18.960999219 +0000 UTC m=+2.992060447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:18.463554 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.463530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbblf\" (UniqueName: \"kubernetes.io/projected/0ade7d6f-8682-4465-a333-a0cc19c9f4e4-kube-api-access-kbblf\") pod \"multus-gffq8\" (UID: \"0ade7d6f-8682-4465-a333-a0cc19c9f4e4\") " pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.464169 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.464145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59j72\" (UniqueName: \"kubernetes.io/projected/91b9e4d0-73d1-4148-be4c-933f45f1ae8c-kube-api-access-59j72\") pod \"iptables-alerter-r9fs8\" (UID: \"91b9e4d0-73d1-4148-be4c-933f45f1ae8c\") " pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.553930 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.553892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-run-netns\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.553941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovnkube-config\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.553965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-kubernetes\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.553986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-kubelet\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.553999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-run-netns\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-env-overrides\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovnkube-script-lib\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-sys\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-socket-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.554124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-kubernetes\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjck\" (UniqueName: \"kubernetes.io/projected/712bb0df-b242-4217-9c6b-f74dad8d4767-kube-api-access-6tjck\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-cni-netd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-device-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-ovn\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovn-node-metrics-cert\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3c2881f-9312-4856-9964-5d0c94db3ea8-hosts-file\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-run\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93fceeb3-bb73-4ec4-956d-553a85b9f92a-host\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-systemd-units\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-var-lib-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nkl\" (UniqueName: \"kubernetes.io/projected/e3c2881f-9312-4856-9964-5d0c94db3ea8-kube-api-access-j9nkl\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysctl-conf\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-etc-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93fceeb3-bb73-4ec4-956d-553a85b9f92a-serviceca\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.554459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-cni-bin\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-system-cni-dir\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-cni-binary-copy\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-sys-fs\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-lib-modules\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-host\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0792af-21ca-43d1-84d3-529bc7965b8c-tmp\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c2881f-9312-4856-9964-5d0c94db3ea8-tmp-dir\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-systemd\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-node-log\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctbb\" (UniqueName: \"kubernetes.io/projected/0068c74d-6fc7-4801-807a-2d9204de6130-kube-api-access-5ctbb\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysctl-d\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtw8h\" (UniqueName: \"kubernetes.io/projected/9c0792af-21ca-43d1-84d3-529bc7965b8c-kube-api-access-wtw8h\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxskl\" (UniqueName: \"kubernetes.io/projected/93fceeb3-bb73-4ec4-956d-553a85b9f92a-kube-api-access-vxskl\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqsd\" (UniqueName: \"kubernetes.io/projected/63bda03e-2063-492a-aba8-2fc3a351bfcf-kube-api-access-bnqsd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysconfig\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovnkube-config\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554784 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-sys\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgclx\" (UniqueName: \"kubernetes.io/projected/3391cf32-789c-4ed8-80b9-65af2d2a6955-kube-api-access-vgclx\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-modprobe-d\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-dbus\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-systemd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-cnibin\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-tuned\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-slash\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-log-socket\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-kubelet-config\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-registration-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.555560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-var-lib-kubelet\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-env-overrides\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.555158 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.555241 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:19.055225607 +0000 UTC m=+3.086286841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-kubelet\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.554719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovnkube-script-lib\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-var-lib-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3c2881f-9312-4856-9964-5d0c94db3ea8-tmp-dir\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-socket-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-node-log\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3c2881f-9312-4856-9964-5d0c94db3ea8-hosts-file\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-run\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/93fceeb3-bb73-4ec4-956d-553a85b9f92a-host\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.556212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-systemd-units\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-systemd\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-cni-bin\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.555972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-system-cni-dir\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysctl-d\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-device-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysctl-conf\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-cni-netd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-ovn\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556274 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0068c74d-6fc7-4801-807a-2d9204de6130-cnibin\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.556296 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:18.556352 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:19.056334451 +0000 UTC m=+3.087395678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-etc-openvswitch\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-cni-binary-copy\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-registration-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.557198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-modprobe-d\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/93fceeb3-bb73-4ec4-956d-553a85b9f92a-serviceca\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-log-socket\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-slash\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-run-systemd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556728 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-host\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/712bb0df-b242-4217-9c6b-f74dad8d4767-sys-fs\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-sysconfig\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-kubelet-config\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-lib-modules\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556837 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-dbus\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c0792af-21ca-43d1-84d3-529bc7965b8c-var-lib-kubelet\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.556863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bda03e-2063-492a-aba8-2fc3a351bfcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.557193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0068c74d-6fc7-4801-807a-2d9204de6130-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.558549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.558267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c0792af-21ca-43d1-84d3-529bc7965b8c-tmp\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.559281 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.559260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63bda03e-2063-492a-aba8-2fc3a351bfcf-ovn-node-metrics-cert\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.559351 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.559328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9c0792af-21ca-43d1-84d3-529bc7965b8c-etc-tuned\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.577290 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.577259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nkl\" (UniqueName: \"kubernetes.io/projected/e3c2881f-9312-4856-9964-5d0c94db3ea8-kube-api-access-j9nkl\") pod \"node-resolver-t5fnc\" (UID: \"e3c2881f-9312-4856-9964-5d0c94db3ea8\") " pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.580573 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.580383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxskl\" (UniqueName: \"kubernetes.io/projected/93fceeb3-bb73-4ec4-956d-553a85b9f92a-kube-api-access-vxskl\") pod \"node-ca-qrnd9\" (UID: \"93fceeb3-bb73-4ec4-956d-553a85b9f92a\") " pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.583004 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.582943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctbb\" (UniqueName: \"kubernetes.io/projected/0068c74d-6fc7-4801-807a-2d9204de6130-kube-api-access-5ctbb\") pod \"multus-additional-cni-plugins-rmxkg\" (UID: \"0068c74d-6fc7-4801-807a-2d9204de6130\") " pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:18.583220 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.583199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjck\" (UniqueName: \"kubernetes.io/projected/712bb0df-b242-4217-9c6b-f74dad8d4767-kube-api-access-6tjck\") pod \"aws-ebs-csi-driver-node-6pdd2\" (UID: \"712bb0df-b242-4217-9c6b-f74dad8d4767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.585155 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.585125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgclx\" (UniqueName: \"kubernetes.io/projected/3391cf32-789c-4ed8-80b9-65af2d2a6955-kube-api-access-vgclx\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:18.585263 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.585233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtw8h\" (UniqueName: \"kubernetes.io/projected/9c0792af-21ca-43d1-84d3-529bc7965b8c-kube-api-access-wtw8h\") pod \"tuned-tspql\" (UID: \"9c0792af-21ca-43d1-84d3-529bc7965b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.586370 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.586346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqsd\" (UniqueName: \"kubernetes.io/projected/63bda03e-2063-492a-aba8-2fc3a351bfcf-kube-api-access-bnqsd\") pod \"ovnkube-node-j545j\" (UID: \"63bda03e-2063-492a-aba8-2fc3a351bfcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.648492 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.648389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:18.659517 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.659487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r9fs8" Sep 11 17:54:18.670536 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.670509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gffq8" Sep 11 17:54:18.679646 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.679619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:18.687899 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.687878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" Sep 11 17:54:18.696055 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.695886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tspql" Sep 11 17:54:18.704864 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.704830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t5fnc" Sep 11 17:54:18.711816 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.711791 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrnd9" Sep 11 17:54:18.718904 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:18.718879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" Sep 11 17:54:19.060484 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:19.060380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:19.060484 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:19.060433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:19.060683 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:19.060495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:19.060683 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060630 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:19.060683 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060634 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:19.060683 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060650 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:19.060811 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:20.060675021 +0000 UTC m=+4.091736235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:19.060811 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060705 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:19.060811 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060715 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:20.060708512 +0000 UTC m=+4.091769722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:19.060811 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060719 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:19.060811 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:19.060766 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:20.060749945 +0000 UTC m=+4.091811173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:19.409123 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:19.409055 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:49:19 +0000 UTC" deadline="2028-05-12 08:35:28.021304182 +0000 UTC" Sep 11 17:54:19.409123 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:19.409117 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14606h41m8.612191379s" Sep 11 17:54:19.535209 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.535170 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c2881f_9312_4856_9964_5d0c94db3ea8.slice/crio-35b7a935ea3faff245aba8b70627ad1a6ca8add91079c5b69db5bf2386512dc3 WatchSource:0}: Error finding container 35b7a935ea3faff245aba8b70627ad1a6ca8add91079c5b69db5bf2386512dc3: Status 404 returned error can't find the container with id 35b7a935ea3faff245aba8b70627ad1a6ca8add91079c5b69db5bf2386512dc3 Sep 11 17:54:19.540060 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.540034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fceeb3_bb73_4ec4_956d_553a85b9f92a.slice/crio-e37a9d75e0ccfa5e2a889fde798193308d92650497ff5a320a4b483700947593 WatchSource:0}: Error finding container e37a9d75e0ccfa5e2a889fde798193308d92650497ff5a320a4b483700947593: Status 404 returned error can't find the container with id e37a9d75e0ccfa5e2a889fde798193308d92650497ff5a320a4b483700947593 Sep 11 17:54:19.541745 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.541714 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bda03e_2063_492a_aba8_2fc3a351bfcf.slice/crio-0c90b6b92062cf93e01e587d322034da9ee9a5d2598570a0289f5ed6984f655b WatchSource:0}: Error finding container 0c90b6b92062cf93e01e587d322034da9ee9a5d2598570a0289f5ed6984f655b: Status 404 returned error can't find the container with id 0c90b6b92062cf93e01e587d322034da9ee9a5d2598570a0289f5ed6984f655b Sep 11 17:54:19.542221 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.542194 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712bb0df_b242_4217_9c6b_f74dad8d4767.slice/crio-0d7f920417d5223f73df364b4a40891adf7e0352a29332a64b39b27a4a1f8234 WatchSource:0}: Error finding container 0d7f920417d5223f73df364b4a40891adf7e0352a29332a64b39b27a4a1f8234: Status 404 returned error can't find the container with id 0d7f920417d5223f73df364b4a40891adf7e0352a29332a64b39b27a4a1f8234 Sep 11 17:54:19.543239 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.543216 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ade7d6f_8682_4465_a333_a0cc19c9f4e4.slice/crio-cd2b9f48d61ca760c305427aa52b55c007f0981fe1f7cd61a01d354a5c0996f1 WatchSource:0}: Error finding container cd2b9f48d61ca760c305427aa52b55c007f0981fe1f7cd61a01d354a5c0996f1: Status 404 returned error can't find the container with id cd2b9f48d61ca760c305427aa52b55c007f0981fe1f7cd61a01d354a5c0996f1 Sep 11 17:54:19.544264 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.543915 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d370f9_8788_4fde_af01_dad42dccac2c.slice/crio-cb1cef8c29aa14b744dd4771388e552a7978c3181bef86085903a696b3c229e3 WatchSource:0}: Error finding container cb1cef8c29aa14b744dd4771388e552a7978c3181bef86085903a696b3c229e3: Status 404 returned error can't find the container with id cb1cef8c29aa14b744dd4771388e552a7978c3181bef86085903a696b3c229e3 Sep 11 17:54:19.545358 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.545336 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b9e4d0_73d1_4148_be4c_933f45f1ae8c.slice/crio-7a54d2458f2d9b94716f9508fd868ac64bd80174d796676d389fb8bd971aa6b1 WatchSource:0}: Error finding container 7a54d2458f2d9b94716f9508fd868ac64bd80174d796676d389fb8bd971aa6b1: Status 404 returned error can't find the container with id 7a54d2458f2d9b94716f9508fd868ac64bd80174d796676d389fb8bd971aa6b1 Sep 11 17:54:19.547856 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.547835 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0068c74d_6fc7_4801_807a_2d9204de6130.slice/crio-cd7f402b0bc2cbbdb7a3619504cbeeae5c3b925a8b2500ec66ae1b31e7d161ce WatchSource:0}: Error finding container cd7f402b0bc2cbbdb7a3619504cbeeae5c3b925a8b2500ec66ae1b31e7d161ce: Status 404 returned error can't find the container with id cd7f402b0bc2cbbdb7a3619504cbeeae5c3b925a8b2500ec66ae1b31e7d161ce Sep 11 17:54:19.549311 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:19.548923 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0792af_21ca_43d1_84d3_529bc7965b8c.slice/crio-3df13ee32721010bd363667e6f85e6bf852ef5e1631aa658cd5c697cd2e566b0 WatchSource:0}: Error finding container 3df13ee32721010bd363667e6f85e6bf852ef5e1631aa658cd5c697cd2e566b0: Status 404 returned error can't find the container with id 3df13ee32721010bd363667e6f85e6bf852ef5e1631aa658cd5c697cd2e566b0 Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.067304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.067653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.067714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.067539 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.067856 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.067867 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.067921 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:22.067906105 +0000 UTC m=+6.098967340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.067987 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.068016 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:22.06800592 +0000 UTC m=+6.099067144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.067833 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:20.068481 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.068051 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:22.068034947 +0000 UTC m=+6.099096170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:20.410472 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.410383 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:49:19 +0000 UTC" deadline="2028-03-25 09:00:47.046062933 +0000 UTC" Sep 11 17:54:20.410472 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.410462 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13455h6m26.635643448s" Sep 11 17:54:20.441382 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.440430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:20.441382 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.440596 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:20.441382 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.441030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:20.441382 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.441144 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:20.441382 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.441216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:20.441382 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:20.441305 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:20.466576 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.466047 2579 generic.go:356] "Generic (PLEG): container finished" podID="3dd652f262855ed205ed8b8367aab7c5" containerID="ef434b3994c0f30a16743e7fb5cb571642de858acd34a99a0595dee6ce77fee7" exitCode=0 Sep 11 17:54:20.466576 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.466155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" event={"ID":"3dd652f262855ed205ed8b8367aab7c5","Type":"ContainerDied","Data":"ef434b3994c0f30a16743e7fb5cb571642de858acd34a99a0595dee6ce77fee7"} Sep 11 17:54:20.474743 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.474715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tspql" event={"ID":"9c0792af-21ca-43d1-84d3-529bc7965b8c","Type":"ContainerStarted","Data":"3df13ee32721010bd363667e6f85e6bf852ef5e1631aa658cd5c697cd2e566b0"} Sep 11 17:54:20.479570 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.479529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6gxrw" event={"ID":"85d370f9-8788-4fde-af01-dad42dccac2c","Type":"ContainerStarted","Data":"cb1cef8c29aa14b744dd4771388e552a7978c3181bef86085903a696b3c229e3"} Sep 11 17:54:20.481791 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.481615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gffq8" event={"ID":"0ade7d6f-8682-4465-a333-a0cc19c9f4e4","Type":"ContainerStarted","Data":"cd2b9f48d61ca760c305427aa52b55c007f0981fe1f7cd61a01d354a5c0996f1"} Sep 11 17:54:20.484668 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.484639 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"0c90b6b92062cf93e01e587d322034da9ee9a5d2598570a0289f5ed6984f655b"} Sep 11 17:54:20.497140 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.497075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrnd9" event={"ID":"93fceeb3-bb73-4ec4-956d-553a85b9f92a","Type":"ContainerStarted","Data":"e37a9d75e0ccfa5e2a889fde798193308d92650497ff5a320a4b483700947593"} Sep 11 17:54:20.513307 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.513268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" event={"ID":"8b998e155b62d16440a868b586601783","Type":"ContainerStarted","Data":"dbbad744c38d378e3251912926d4ad0d13eec39ac432c536d431d336c5363199"} Sep 11 17:54:20.539783 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.539689 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-255.ec2.internal" podStartSLOduration=2.539667948 podStartE2EDuration="2.539667948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:20.539186197 +0000 UTC m=+4.570247456" watchObservedRunningTime="2026-09-11 17:54:20.539667948 +0000 UTC m=+4.570729184" Sep 11 17:54:20.554659 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.554594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerStarted","Data":"cd7f402b0bc2cbbdb7a3619504cbeeae5c3b925a8b2500ec66ae1b31e7d161ce"} Sep 11 17:54:20.582719 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.582663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r9fs8" event={"ID":"91b9e4d0-73d1-4148-be4c-933f45f1ae8c","Type":"ContainerStarted","Data":"7a54d2458f2d9b94716f9508fd868ac64bd80174d796676d389fb8bd971aa6b1"} Sep 11 17:54:20.593074 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.592967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" event={"ID":"712bb0df-b242-4217-9c6b-f74dad8d4767","Type":"ContainerStarted","Data":"0d7f920417d5223f73df364b4a40891adf7e0352a29332a64b39b27a4a1f8234"} Sep 11 17:54:20.614919 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:20.614831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t5fnc" event={"ID":"e3c2881f-9312-4856-9964-5d0c94db3ea8","Type":"ContainerStarted","Data":"35b7a935ea3faff245aba8b70627ad1a6ca8add91079c5b69db5bf2386512dc3"} Sep 11 17:54:21.644582 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:21.644533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" event={"ID":"3dd652f262855ed205ed8b8367aab7c5","Type":"ContainerStarted","Data":"8df2bb56aa05bf3d1e36b0e23cdeca30511a2fc5c75852448d5ecc51cf29789f"} Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:22.085393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:22.085482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:22.085527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.085691 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.085764 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:26.085742832 +0000 UTC m=+10.116804066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.086188 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.086242 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:26.086223272 +0000 UTC m=+10.117284496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.086327 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.086346 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.086358 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:22.086534 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.086399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:26.086385426 +0000 UTC m=+10.117446657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:22.439270 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:22.438553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:22.439270 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.438667 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:22.439270 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:22.439059 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:22.439270 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.439155 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:22.442557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:22.439710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:22.442557 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:22.439828 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:24.439808 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:24.439577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:24.439808 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:24.439602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:24.439808 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:24.439654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:24.439808 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:24.439744 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:24.440367 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:24.439841 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:24.440367 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:24.439912 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:26.120147 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:26.120100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:26.120648 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:26.120175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:26.120648 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:26.120208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:26.120648 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.120341 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:26.120648 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.120407 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:34.12038676 +0000 UTC m=+18.151447989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:26.120877 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.120862 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:26.120935 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.120921 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:34.120902656 +0000 UTC m=+18.151963867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:26.121030 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.121009 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:26.121030 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.121035 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:26.121260 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.121047 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:26.121260 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.121086 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:34.121073547 +0000 UTC m=+18.152134762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:26.441085 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:26.441011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:26.441240 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:26.441210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:26.441348 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.441195 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:26.441414 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.441349 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:26.441414 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:26.441391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:26.441541 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:26.441492 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:28.438482 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:28.438024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:28.438482 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:28.438174 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:28.438915 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:28.438505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:28.438915 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:28.438628 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:28.439059 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:28.439033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:28.439194 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:28.439150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:30.438675 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:30.438631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:30.439021 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:30.438637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:30.439021 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:30.438766 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:30.439021 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:30.438867 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:30.439021 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:30.438637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:30.439021 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:30.438964 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:32.437648 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:32.437620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:32.438096 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:32.437662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:32.438096 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:32.437738 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:32.438096 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:32.437629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:32.438096 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:32.437886 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:32.438096 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:32.437969 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:34.178336 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:34.178300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:34.178363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:34.178410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178464 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178533 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:50.178514435 +0000 UTC m=+34.209575666 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178551 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178554 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178580 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178593 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178612 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:50.17859401 +0000 UTC m=+34.209655245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:34.178886 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.178635 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:50.178622125 +0000 UTC m=+34.209683351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:34.438801 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:34.438706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:34.438950 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:34.438708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:34.438950 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.438845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:34.438950 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:34.438857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:34.439066 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.438958 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:34.439110 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:34.439066 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:36.439322 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:36.439288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:36.439814 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:36.439388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:36.439814 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:36.439383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:36.439814 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:36.439514 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:36.439814 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:36.439549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:36.439814 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:36.439653 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:38.442250 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.442218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:38.442592 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:38.442353 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:38.442592 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.442495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:38.442592 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:38.442570 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:38.442920 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.442899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:38.443063 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:38.443020 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:38.683454 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.682914 2579 generic.go:356] "Generic (PLEG): container finished" podID="0068c74d-6fc7-4801-807a-2d9204de6130" containerID="bbaf9c95437ae73d3a541bebb53d641ec087ba007cee80676bd1e4913118ae2c" exitCode=0 Sep 11 17:54:38.683597 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.682993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerDied","Data":"bbaf9c95437ae73d3a541bebb53d641ec087ba007cee80676bd1e4913118ae2c"} Sep 11 17:54:38.685486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.685459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" event={"ID":"712bb0df-b242-4217-9c6b-f74dad8d4767","Type":"ContainerStarted","Data":"ca5233e9b165e8400b32f2d9d216c85fed82450f9f42a2597d2e72b7368ad1f9"} Sep 11 17:54:38.688096 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.688068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t5fnc" event={"ID":"e3c2881f-9312-4856-9964-5d0c94db3ea8","Type":"ContainerStarted","Data":"4957d039a12af24a3f151ef5eb3fb936acacfb135f6e08d9ebf6b8a9f7f90ffc"} Sep 11 17:54:38.690059 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.689678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tspql" event={"ID":"9c0792af-21ca-43d1-84d3-529bc7965b8c","Type":"ContainerStarted","Data":"fc9aae39ba0de6dfb76490a8b4306f45373e218e815d608e21d91e05447210bb"} Sep 11 17:54:38.691489 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.691469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6gxrw" event={"ID":"85d370f9-8788-4fde-af01-dad42dccac2c","Type":"ContainerStarted","Data":"baac7ae503d6e57dc9e46064a6e4a766dfb14e17ea0b0977d61602260b927389"} Sep 11 17:54:38.693120 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.693101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gffq8" event={"ID":"0ade7d6f-8682-4465-a333-a0cc19c9f4e4","Type":"ContainerStarted","Data":"eab32bf91d1d696ce4234afbdd60e660473e1360ba674a088d7f4a31b5dbdfae"} Sep 11 17:54:38.695052 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.695027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"aebb46143307405ef3b49161ba0d24f58ec25463673aad10060ee46028c6e279"} Sep 11 17:54:38.695132 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.695088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"473ae6dc6f4c9f451b084c44b2a91b20e4ceea1a4976a9d9f1d0999bd34597db"} Sep 11 17:54:38.696628 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.696608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrnd9" event={"ID":"93fceeb3-bb73-4ec4-956d-553a85b9f92a","Type":"ContainerStarted","Data":"1da447a647a50e0e5eafdb4339a596fa1deeefd5c832e030bd3247e3d988f11c"} Sep 11 17:54:38.712211 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.712170 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-255.ec2.internal" podStartSLOduration=20.712157804 podStartE2EDuration="20.712157804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:21.664305258 +0000 UTC m=+5.695366493" watchObservedRunningTime="2026-09-11 17:54:38.712157804 +0000 UTC m=+22.743219033" Sep 11 17:54:38.729087 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.729048 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6gxrw" podStartSLOduration=6.6265207969999995 podStartE2EDuration="21.729035442s" totalImagesPullingTime="15.102514645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:38.728878012 +0000 UTC m=+22.759939251" watchObservedRunningTime="2026-09-11 17:54:38.729035442 +0000 UTC m=+22.760096672" Sep 11 17:54:38.749975 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.749905 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tspql" podStartSLOduration=2.927099706 podStartE2EDuration="21.74988512s" totalImagesPullingTime="18.822785414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:38.749083077 +0000 UTC m=+22.780144307" watchObservedRunningTime="2026-09-11 17:54:38.74988512 +0000 UTC m=+22.780946351" Sep 11 17:54:38.771222 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.771185 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t5fnc" podStartSLOduration=2.961091018 podStartE2EDuration="21.771171495s" totalImagesPullingTime="18.810080477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:38.770623475 +0000 UTC m=+22.801684705" watchObservedRunningTime="2026-09-11 17:54:38.771171495 +0000 UTC m=+22.802232726" Sep 11 17:54:38.795093 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.795041 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gffq8" podStartSLOduration=2.928012454 podStartE2EDuration="21.795027696s" totalImagesPullingTime="18.867015242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:38.794841455 +0000 UTC m=+22.825902685" watchObservedRunningTime="2026-09-11 17:54:38.795027696 +0000 UTC m=+22.826088926" Sep 11 17:54:38.810695 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.810651 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrnd9" podStartSLOduration=3.005810472 podStartE2EDuration="21.810633005s" totalImagesPullingTime="18.804822533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:38.810456465 +0000 UTC m=+22.841517707" watchObservedRunningTime="2026-09-11 17:54:38.810633005 +0000 UTC m=+22.841694236" Sep 11 17:54:38.952901 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:38.952876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:39.026564 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:39.026533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:39.703143 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:39.703005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"4376d219f70b9a24b418551a201067e5d5e02de0364dc5dfcedb4b3ad484b3f1"} Sep 11 17:54:39.703143 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:39.703133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"77c9e3630826324c892eb5c9f57a63950106fd915d89867aa1a56f796ae7067b"} Sep 11 17:54:39.704083 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:39.703190 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"0333a57631ffd353a1e81313715b81ce571f9dbbaca295f72c06715a0b8b4ca3"} Sep 11 17:54:39.704083 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:39.703238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"58dc59a7b9dd7287cb0230ee7f109bd1494597675f317544234543784388c5fa"} Sep 11 17:54:40.117347 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.117306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:40.196828 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.196799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:40.340999 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.340959 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 17:54:40.416973 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.416792 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:54:40.340984089Z","UUID":"88f54e9c-6878-4b42-906b-8583effb09ee","Handler":null,"Name":"","Endpoint":""} Sep 11 17:54:40.418201 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.418183 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 17:54:40.418304 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.418211 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 17:54:40.441222 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.441196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:40.441356 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.441288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:40.441356 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:40.441294 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:40.441356 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.441314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:40.441695 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:40.441648 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:40.441943 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:40.441921 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:40.706215 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.706129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r9fs8" event={"ID":"91b9e4d0-73d1-4148-be4c-933f45f1ae8c","Type":"ContainerStarted","Data":"6ccb5c8f2f05d82483d7e9b79504937b9e9cbc24a002786c808b53127c4bf6ea"} Sep 11 17:54:40.707946 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.707922 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" event={"ID":"712bb0df-b242-4217-9c6b-f74dad8d4767","Type":"ContainerStarted","Data":"f35c1af32ae4ae68be3285db09304c6c56d10d1e329ad8d0b76dd642b085ed6e"} Sep 11 17:54:40.728884 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.728838 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r9fs8" podStartSLOduration=4.928180043 podStartE2EDuration="23.728825812s" totalImagesPullingTime="18.800645769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:40.728719757 +0000 UTC m=+24.759780987" watchObservedRunningTime="2026-09-11 17:54:40.728825812 +0000 UTC m=+24.759887041" Sep 11 17:54:40.859574 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.859540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:40.983479 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:40.983382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:41.107807 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:41.107771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:41.712575 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:41.712535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" event={"ID":"712bb0df-b242-4217-9c6b-f74dad8d4767","Type":"ContainerStarted","Data":"f005da640b1967b90a164e5572d4391051511ff7fa06080db2a00e36cc0bb308"} Sep 11 17:54:41.716775 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:41.716746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"a32ebeb376646bf08f86a59cd47bb3766f254c69f8191339364905232c5a6591"} Sep 11 17:54:41.733700 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:41.733645 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdd2" podStartSLOduration=3.386907468 podStartE2EDuration="24.733628323s" totalImagesPullingTime="21.346720855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:41.733115133 +0000 UTC m=+25.764176382" watchObservedRunningTime="2026-09-11 17:54:41.733628323 +0000 UTC m=+25.764689553" Sep 11 17:54:41.755216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:41.755189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:41.755758 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:41.755740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:42.441084 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.441046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:42.441494 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.441473 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:42.441613 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.441499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:42.441613 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.441499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:42.441613 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:42.441570 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:42.441747 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:42.441649 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:42.441747 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:42.441702 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:42.540429 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.540375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:42.959881 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.959841 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:42.960804 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:42.960590 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6gxrw" Sep 11 17:54:43.066685 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:43.066654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:44.438807 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.438635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:44.439451 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.438637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:44.439451 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:44.438873 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:44.439451 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:44.438987 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:44.439451 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.438702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:44.439451 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:44.439088 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:44.727379 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.727290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" event={"ID":"63bda03e-2063-492a-aba8-2fc3a351bfcf","Type":"ContainerStarted","Data":"e91f27c9ca45181e7eed59b8434710a4c99591de3c1b35706dd8639452ff0619"} Sep 11 17:54:44.727665 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.727638 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:44.729178 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.729152 2579 generic.go:356] "Generic (PLEG): container finished" podID="0068c74d-6fc7-4801-807a-2d9204de6130" containerID="1cbf819e10f050ea842906bcaf8d4360f9290d181650b5512ef6716b2cf9aaf7" exitCode=0 Sep 11 17:54:44.729278 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.729240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerDied","Data":"1cbf819e10f050ea842906bcaf8d4360f9290d181650b5512ef6716b2cf9aaf7"} Sep 11 17:54:44.744205 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.744177 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:44.770357 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:44.770312 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" podStartSLOduration=8.901153929 podStartE2EDuration="27.770298214s" totalImagesPullingTime="18.869144285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:44.770180308 +0000 UTC m=+28.801241537" watchObservedRunningTime="2026-09-11 17:54:44.770298214 +0000 UTC m=+28.801359443" Sep 11 17:54:45.734913 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.734664 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:45.735364 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.735016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:45.754880 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.754851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:54:45.951891 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.951864 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sscnw"] Sep 11 17:54:45.952040 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.951983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:45.952098 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:45.952077 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:45.952354 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.952335 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z99jr"] Sep 11 17:54:45.952450 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.952437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:45.952542 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:45.952527 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:45.956064 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.956039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8wml"] Sep 11 17:54:45.956158 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:45.956137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:45.956242 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:45.956222 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:46.736199 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:46.736160 2579 generic.go:356] "Generic (PLEG): container finished" podID="0068c74d-6fc7-4801-807a-2d9204de6130" containerID="29191d97cb6760fdee05470e7398f3779275297a3222260431d39f982853f7cc" exitCode=0 Sep 11 17:54:46.736714 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:46.736250 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerDied","Data":"29191d97cb6760fdee05470e7398f3779275297a3222260431d39f982853f7cc"} Sep 11 17:54:47.438537 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:47.438504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:47.438704 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:47.438504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:47.438704 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:47.438639 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:47.438832 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:47.438742 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:47.438832 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:47.438512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:47.438916 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:47.438834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:48.742413 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:48.742224 2579 generic.go:356] "Generic (PLEG): container finished" podID="0068c74d-6fc7-4801-807a-2d9204de6130" containerID="75de773dd7ff5c377ad704a5447621c21ed38d03271e18937d523426953d9cb9" exitCode=0 Sep 11 17:54:48.742413 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:48.742307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerDied","Data":"75de773dd7ff5c377ad704a5447621c21ed38d03271e18937d523426953d9cb9"} Sep 11 17:54:49.438530 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:49.438502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:49.438530 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:49.438521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:49.438760 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:49.438627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:49.438760 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:49.438641 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:49.438760 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:49.438695 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:49.438919 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:49.438773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:50.191705 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:50.191674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:50.191705 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:50.191717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:50.191751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191837 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191837 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191879 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191900 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret podName:e12fa326-e7dc-47bf-93d9-dd1f0344a2c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:22.191881003 +0000 UTC m=+66.222942225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret") pod "global-pull-secret-syncer-z99jr" (UID: "e12fa326-e7dc-47bf-93d9-dd1f0344a2c5") : object "kube-system"/"original-pull-secret" not registered Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191904 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191917 2579 projected.go:196] Error preparing data for projected volume kube-api-access-6tz92 for pod openshift-network-diagnostics/network-check-target-t8wml: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs podName:3391cf32-789c-4ed8-80b9-65af2d2a6955 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:22.191914277 +0000 UTC m=+66.222975486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs") pod "network-metrics-daemon-sscnw" (UID: "3391cf32-789c-4ed8-80b9-65af2d2a6955") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 17:54:50.192166 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:50.191973 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92 podName:0383e6d5-6477-44cf-b7a0-3d00c9d55472 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:22.19195597 +0000 UTC m=+66.223017196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6tz92" (UniqueName: "kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92") pod "network-check-target-t8wml" (UID: "0383e6d5-6477-44cf-b7a0-3d00c9d55472") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 17:54:51.438165 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:51.438133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:51.438165 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:51.438165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:51.438601 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:51.438168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:51.438601 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:51.438236 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sscnw" podUID="3391cf32-789c-4ed8-80b9-65af2d2a6955" Sep 11 17:54:51.438601 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:51.438312 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z99jr" podUID="e12fa326-e7dc-47bf-93d9-dd1f0344a2c5" Sep 11 17:54:51.438601 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:51.438371 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8wml" podUID="0383e6d5-6477-44cf-b7a0-3d00c9d55472" Sep 11 17:54:52.263569 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.263497 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-255.ec2.internal" event="NodeReady" Sep 11 17:54:52.263722 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.263667 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 17:54:52.316058 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.315184 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wlz59"] Sep 11 17:54:52.320564 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.320539 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9pnbx"] Sep 11 17:54:52.320686 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.320670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.324354 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.324008 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 17:54:52.324354 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.324010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.324354 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.324158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cj7w5\"" Sep 11 17:54:52.324649 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.324630 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 17:54:52.327631 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.327615 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 17:54:52.327781 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.327764 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 17:54:52.327888 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.327867 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 17:54:52.328263 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.328059 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7qbb\"" Sep 11 17:54:52.331713 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.331696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wlz59"] Sep 11 17:54:52.332249 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.332227 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9pnbx"] Sep 11 17:54:52.407465 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.407433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.407633 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.407471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522b462e-cb2f-4b6a-9a16-d84758954478-tmp-dir\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.407633 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.407499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8b2r\" (UniqueName: \"kubernetes.io/projected/522b462e-cb2f-4b6a-9a16-d84758954478-kube-api-access-z8b2r\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.407633 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.407526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b8b095b-591c-4f3d-97c6-bc7b5833ab38-cert\") pod \"ingress-canary-9pnbx\" (UID: \"4b8b095b-591c-4f3d-97c6-bc7b5833ab38\") " pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.407633 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.407595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/522b462e-cb2f-4b6a-9a16-d84758954478-config-volume\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.407806 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.407653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgp8\" (UniqueName: \"kubernetes.io/projected/4b8b095b-591c-4f3d-97c6-bc7b5833ab38-kube-api-access-dtgp8\") pod \"ingress-canary-9pnbx\" (UID: \"4b8b095b-591c-4f3d-97c6-bc7b5833ab38\") " pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.465438 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.465396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:52.508435 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgp8\" (UniqueName: \"kubernetes.io/projected/4b8b095b-591c-4f3d-97c6-bc7b5833ab38-kube-api-access-dtgp8\") pod \"ingress-canary-9pnbx\" (UID: \"4b8b095b-591c-4f3d-97c6-bc7b5833ab38\") " pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.508615 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.508615 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522b462e-cb2f-4b6a-9a16-d84758954478-tmp-dir\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.508615 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8b2r\" (UniqueName: \"kubernetes.io/projected/522b462e-cb2f-4b6a-9a16-d84758954478-kube-api-access-z8b2r\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.508615 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b8b095b-591c-4f3d-97c6-bc7b5833ab38-cert\") pod \"ingress-canary-9pnbx\" (UID: \"4b8b095b-591c-4f3d-97c6-bc7b5833ab38\") " pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.508615 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/522b462e-cb2f-4b6a-9a16-d84758954478-config-volume\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.508855 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:52.508611 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 17:54:52.508855 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:52.508679 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls podName:522b462e-cb2f-4b6a-9a16-d84758954478 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:53.008658821 +0000 UTC m=+37.039720049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls") pod "dns-default-wlz59" (UID: "522b462e-cb2f-4b6a-9a16-d84758954478") : secret "dns-default-metrics-tls" not found Sep 11 17:54:52.508989 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.508931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522b462e-cb2f-4b6a-9a16-d84758954478-tmp-dir\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.509190 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.509169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/522b462e-cb2f-4b6a-9a16-d84758954478-config-volume\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.513355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.513335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b8b095b-591c-4f3d-97c6-bc7b5833ab38-cert\") pod \"ingress-canary-9pnbx\" (UID: \"4b8b095b-591c-4f3d-97c6-bc7b5833ab38\") " pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.518114 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.518057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgp8\" (UniqueName: \"kubernetes.io/projected/4b8b095b-591c-4f3d-97c6-bc7b5833ab38-kube-api-access-dtgp8\") pod \"ingress-canary-9pnbx\" (UID: \"4b8b095b-591c-4f3d-97c6-bc7b5833ab38\") " pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:52.518200 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.518174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8b2r\" (UniqueName: \"kubernetes.io/projected/522b462e-cb2f-4b6a-9a16-d84758954478-kube-api-access-z8b2r\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:52.568454 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.568427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:52.638720 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:52.638688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9pnbx" Sep 11 17:54:53.012921 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.012881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:53.013104 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:53.013053 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 17:54:53.013167 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:53.013119 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls podName:522b462e-cb2f-4b6a-9a16-d84758954478 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:54.013104099 +0000 UTC m=+38.044165321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls") pod "dns-default-wlz59" (UID: "522b462e-cb2f-4b6a-9a16-d84758954478") : secret "dns-default-metrics-tls" not found Sep 11 17:54:53.315641 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.315455 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9pnbx"] Sep 11 17:54:53.338323 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:54:53.338292 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8b095b_591c_4f3d_97c6_bc7b5833ab38.slice/crio-0b5e73a5452d6065375ea1c8d38393b68f6c019c5d5111c11913f084437d77d5 WatchSource:0}: Error finding container 0b5e73a5452d6065375ea1c8d38393b68f6c019c5d5111c11913f084437d77d5: Status 404 returned error can't find the container with id 0b5e73a5452d6065375ea1c8d38393b68f6c019c5d5111c11913f084437d77d5 Sep 11 17:54:53.438256 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.438222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:54:53.438256 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.438243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:54:53.438472 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.438222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:54:53.443594 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.443564 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 17:54:53.444240 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.444223 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhttf\"" Sep 11 17:54:53.444240 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.444235 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-64859\"" Sep 11 17:54:53.445084 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.444912 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 17:54:53.445084 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.444994 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 17:54:53.445084 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.444994 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 17:54:53.757303 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.757263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9pnbx" event={"ID":"4b8b095b-591c-4f3d-97c6-bc7b5833ab38","Type":"ContainerStarted","Data":"0b5e73a5452d6065375ea1c8d38393b68f6c019c5d5111c11913f084437d77d5"} Sep 11 17:54:53.763141 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.763110 2579 generic.go:356] "Generic (PLEG): container finished" podID="0068c74d-6fc7-4801-807a-2d9204de6130" containerID="3c851ed60e4ccda6bf6b93891e8af28d6bf2b2a757bf92fb03cff5472d653be8" exitCode=0 Sep 11 17:54:53.763272 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.763160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerDied","Data":"3c851ed60e4ccda6bf6b93891e8af28d6bf2b2a757bf92fb03cff5472d653be8"} Sep 11 17:54:53.794670 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.794644 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64477965b8-nbp56"] Sep 11 17:54:53.798847 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.798830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:53.802325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802083 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 17:54:53.802325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802106 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 17:54:53.802325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bj5dv\"" Sep 11 17:54:53.802325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802083 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 17:54:53.802325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802143 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 17:54:53.802636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 17:54:53.802636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.802492 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 17:54:53.809992 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.809970 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64477965b8-nbp56"] Sep 11 17:54:53.919740 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.919682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-default-certificate\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:53.919849 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.919795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:53.919913 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.919848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:53.919913 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.919867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsf6\" (UniqueName: \"kubernetes.io/projected/66738241-d01b-4e83-9c45-1be1afac1e1e-kube-api-access-tvsf6\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:53.919913 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.919896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-stats-auth\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:53.954786 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:53.954758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:54.020917 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.020833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-default-certificate\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.020917 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.020885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.021133 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.020921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:54.021133 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.020968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.021133 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.020991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsf6\" (UniqueName: \"kubernetes.io/projected/66738241-d01b-4e83-9c45-1be1afac1e1e-kube-api-access-tvsf6\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.021133 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.021106 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 17:54:54.021329 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.021140 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:54.521115102 +0000 UTC m=+38.552176327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : configmap references non-existent config key: service-ca.crt Sep 11 17:54:54.021329 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.021171 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:54.521159295 +0000 UTC m=+38.552220506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : secret "router-metrics-certs-default" not found Sep 11 17:54:54.021329 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.021186 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 17:54:54.021329 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.021237 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls podName:522b462e-cb2f-4b6a-9a16-d84758954478 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:56.021220733 +0000 UTC m=+40.052281946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls") pod "dns-default-wlz59" (UID: "522b462e-cb2f-4b6a-9a16-d84758954478") : secret "dns-default-metrics-tls" not found Sep 11 17:54:54.021329 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.021260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-stats-auth\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.024787 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.024765 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-stats-auth\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.024906 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.024820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-default-certificate\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.030519 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.030486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsf6\" (UniqueName: \"kubernetes.io/projected/66738241-d01b-4e83-9c45-1be1afac1e1e-kube-api-access-tvsf6\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.042301 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.042277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:54.526097 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.526037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.526266 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.526122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:54.526266 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.526236 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:55.526211876 +0000 UTC m=+39.557273109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : configmap references non-existent config key: service-ca.crt Sep 11 17:54:54.526408 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.526295 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 17:54:54.526408 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:54.526354 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:55.526338861 +0000 UTC m=+39.557400084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : secret "router-metrics-certs-default" not found Sep 11 17:54:54.771685 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.771647 2579 generic.go:356] "Generic (PLEG): container finished" podID="0068c74d-6fc7-4801-807a-2d9204de6130" containerID="a32e9668cbfdac6f2cfab301126cb145898eb7ffcc4af22c3133b4c600ce0d03" exitCode=0 Sep 11 17:54:54.772088 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:54.771706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerDied","Data":"a32e9668cbfdac6f2cfab301126cb145898eb7ffcc4af22c3133b4c600ce0d03"} Sep 11 17:54:55.533917 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:55.533877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:55.534109 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:55.533941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:55.534109 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:55.534072 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 17:54:55.534109 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:55.534080 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:57.534058623 +0000 UTC m=+41.565119832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : configmap references non-existent config key: service-ca.crt Sep 11 17:54:55.534276 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:55.534130 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:57.534112861 +0000 UTC m=+41.565174068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : secret "router-metrics-certs-default" not found Sep 11 17:54:55.777244 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:55.777168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" event={"ID":"0068c74d-6fc7-4801-807a-2d9204de6130","Type":"ContainerStarted","Data":"5ffa82714058f65e6271a951bfd3a50f1808961f1669fcc5545f34e34cc9fe97"} Sep 11 17:54:55.821074 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:55.821026 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rmxkg" podStartSLOduration=7.794360496 podStartE2EDuration="38.821011651s" totalImagesPullingTime="31.026651155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:55.820672471 +0000 UTC m=+39.851733700" watchObservedRunningTime="2026-09-11 17:54:55.821011651 +0000 UTC m=+39.852072882" Sep 11 17:54:55.963988 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:55.963966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:56.037890 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:56.037819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:54:56.038027 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:56.037958 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 17:54:56.038027 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:56.038016 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls podName:522b462e-cb2f-4b6a-9a16-d84758954478 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:00.038001887 +0000 UTC m=+44.069063095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls") pod "dns-default-wlz59" (UID: "522b462e-cb2f-4b6a-9a16-d84758954478") : secret "dns-default-metrics-tls" not found Sep 11 17:54:56.111495 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:56.111458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:56.781507 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:56.781472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9pnbx" event={"ID":"4b8b095b-591c-4f3d-97c6-bc7b5833ab38","Type":"ContainerStarted","Data":"6e4642156a945222ed9680974d4c9d5387bfdabc129ae9bc82b31085e0b8d9da"} Sep 11 17:54:56.799712 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:56.799596 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9pnbx" podStartSLOduration=2.232315749 podStartE2EDuration="4.799578219s" totalImagesPullingTime="2.56726247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:56.799272947 +0000 UTC m=+40.830334188" watchObservedRunningTime="2026-09-11 17:54:56.799578219 +0000 UTC m=+40.830639450" Sep 11 17:54:57.084440 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.084348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:57.158722 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.158692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:57.548158 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.548126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:57.548326 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.548170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:54:57.548326 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.548251 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 17:54:57.548326 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.548311 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.548290219 +0000 UTC m=+45.579351433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : configmap references non-existent config key: service-ca.crt Sep 11 17:54:57.548544 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.548340 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.548327673 +0000 UTC m=+45.579388882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : secret "router-metrics-certs-default" not found Sep 11 17:54:57.666055 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.666029 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5zn7n"] Sep 11 17:54:57.674651 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.674621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:57.688542 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.688518 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5zn7n"] Sep 11 17:54:57.688671 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.688635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.691516 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.691487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 17:54:57.691709 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.691695 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 17:54:57.691876 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.691858 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 17:54:57.691959 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.691948 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntvnk\"" Sep 11 17:54:57.692216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.692202 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 17:54:57.743336 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.743310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:54:57.850132 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.850039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.850132 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.850088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.850530 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.850140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0cf0486-0899-4578-abb2-3667f19ad281-crio-socket\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.850530 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.850173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0486-0899-4578-abb2-3667f19ad281-data-volume\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.850530 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.850254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtsx\" (UniqueName: \"kubernetes.io/projected/c0cf0486-0899-4578-abb2-3667f19ad281-kube-api-access-9vtsx\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951064 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtsx\" (UniqueName: \"kubernetes.io/projected/c0cf0486-0899-4578-abb2-3667f19ad281-kube-api-access-9vtsx\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951229 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951266 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951339 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0cf0486-0899-4578-abb2-3667f19ad281-crio-socket\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951339 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0486-0899-4578-abb2-3667f19ad281-data-volume\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951467 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.951367 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 17:54:57.951467 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.951364 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 17:54:57.951557 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.951473 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:58.451450897 +0000 UTC m=+42.482512126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : secret "insights-runtime-extractor-tls" not found Sep 11 17:54:57.951557 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:57.951503 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:58.451487885 +0000 UTC m=+42.482549096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : configmap "kube-rbac-proxy" not found Sep 11 17:54:57.951557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951536 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0cf0486-0899-4578-abb2-3667f19ad281-crio-socket\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.951706 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.951637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0486-0899-4578-abb2-3667f19ad281-data-volume\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:57.961274 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:57.961250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtsx\" (UniqueName: \"kubernetes.io/projected/c0cf0486-0899-4578-abb2-3667f19ad281-kube-api-access-9vtsx\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:58.454835 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:58.454802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:58.454835 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:58.454838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:58.455091 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:58.454911 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 17:54:58.455091 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:58.454968 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:59.454955417 +0000 UTC m=+43.486016639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : configmap "kube-rbac-proxy" not found Sep 11 17:54:58.455091 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:58.454982 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 17:54:58.455091 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:58.455043 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:59.455025923 +0000 UTC m=+43.486087149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : secret "insights-runtime-extractor-tls" not found Sep 11 17:54:59.463042 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:59.463001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:59.463042 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:59.463045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:54:59.463478 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:59.463125 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 17:54:59.463478 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:59.463173 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 17:54:59.463478 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:59.463177 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.463164524 +0000 UTC m=+45.494225746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : configmap "kube-rbac-proxy" not found Sep 11 17:54:59.463478 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:54:59.463240 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.463221266 +0000 UTC m=+45.494282474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : secret "insights-runtime-extractor-tls" not found Sep 11 17:54:59.812617 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:54:59.812545 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5fnc_e3c2881f-9312-4856-9964-5d0c94db3ea8/dns-node-resolver/0.log" Sep 11 17:55:00.068788 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:00.068715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:00.068914 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:00.068860 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 17:55:00.068948 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:00.068922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls podName:522b462e-cb2f-4b6a-9a16-d84758954478 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:08.068906426 +0000 UTC m=+52.099967638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls") pod "dns-default-wlz59" (UID: "522b462e-cb2f-4b6a-9a16-d84758954478") : secret "dns-default-metrics-tls" not found Sep 11 17:55:00.613529 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:00.613501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrnd9_93fceeb3-bb73-4ec4-956d-553a85b9f92a/node-ca/0.log" Sep 11 17:55:01.013330 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:01.013305 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9pnbx_4b8b095b-591c-4f3d-97c6-bc7b5833ab38/serve-healthcheck-canary/0.log" Sep 11 17:55:01.478994 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:01.478955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:01.478994 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:01.478997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:01.479194 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.479105 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 17:55:01.479194 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.479167 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:05.479152612 +0000 UTC m=+49.510213821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : secret "insights-runtime-extractor-tls" not found Sep 11 17:55:01.479269 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.479109 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 17:55:01.479269 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.479245 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:05.479234021 +0000 UTC m=+49.510295229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : configmap "kube-rbac-proxy" not found Sep 11 17:55:01.580331 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:01.580293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:01.580468 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:01.580349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:01.580509 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.580479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:09.580460373 +0000 UTC m=+53.611521596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : configmap references non-existent config key: service-ca.crt Sep 11 17:55:01.580509 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.580499 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 17:55:01.580580 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:01.580539 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:09.580527883 +0000 UTC m=+53.611589090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : secret "router-metrics-certs-default" not found Sep 11 17:55:02.099288 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:02.099257 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:02.183052 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:02.183019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:03.170815 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.170784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:03.316061 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.316028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:03.796506 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.796475 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r"] Sep 11 17:55:03.827599 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.827563 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r"] Sep 11 17:55:03.827754 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.827691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:03.830935 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.830903 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 17:55:03.830935 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.830920 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 17:55:03.830935 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.830923 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 17:55:03.832400 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.832382 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mqjzn\"" Sep 11 17:55:03.832543 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.832434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 17:55:03.998664 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.998630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ace9b19-c060-4771-b195-d1c1d4687584-config\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:03.998832 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.998683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k57xr\" (UniqueName: \"kubernetes.io/projected/4ace9b19-c060-4771-b195-d1c1d4687584-kube-api-access-k57xr\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:03.998832 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:03.998777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ace9b19-c060-4771-b195-d1c1d4687584-serving-cert\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.099298 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.099223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ace9b19-c060-4771-b195-d1c1d4687584-config\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.099298 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.099277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k57xr\" (UniqueName: \"kubernetes.io/projected/4ace9b19-c060-4771-b195-d1c1d4687584-kube-api-access-k57xr\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.099489 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.099466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ace9b19-c060-4771-b195-d1c1d4687584-serving-cert\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.099874 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.099843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ace9b19-c060-4771-b195-d1c1d4687584-config\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.101729 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.101706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ace9b19-c060-4771-b195-d1c1d4687584-serving-cert\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.108536 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.108518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k57xr\" (UniqueName: \"kubernetes.io/projected/4ace9b19-c060-4771-b195-d1c1d4687584-kube-api-access-k57xr\") pod \"service-ca-operator-cf7c776b7-kxr2r\" (UID: \"4ace9b19-c060-4771-b195-d1c1d4687584\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.136634 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.136595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" Sep 11 17:55:04.420946 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.420923 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r"] Sep 11 17:55:04.422704 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:04.422675 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ace9b19_c060_4771_b195_d1c1d4687584.slice/crio-279d67ad547ab395f2109901f9300623cd4f5867366abfd46f81ba74bfdbfad0 WatchSource:0}: Error finding container 279d67ad547ab395f2109901f9300623cd4f5867366abfd46f81ba74bfdbfad0: Status 404 returned error can't find the container with id 279d67ad547ab395f2109901f9300623cd4f5867366abfd46f81ba74bfdbfad0 Sep 11 17:55:04.800098 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:04.800012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" event={"ID":"4ace9b19-c060-4771-b195-d1c1d4687584","Type":"ContainerStarted","Data":"279d67ad547ab395f2109901f9300623cd4f5867366abfd46f81ba74bfdbfad0"} Sep 11 17:55:05.511428 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:05.511374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:05.511885 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:05.511447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:05.511885 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:05.511537 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 17:55:05.511885 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:05.511587 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 17:55:05.511885 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:05.511602 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:13.511581232 +0000 UTC m=+57.542642463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : configmap "kube-rbac-proxy" not found Sep 11 17:55:05.511885 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:05.511668 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:13.511647767 +0000 UTC m=+57.542708983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : secret "insights-runtime-extractor-tls" not found Sep 11 17:55:07.809338 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:07.809306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" event={"ID":"4ace9b19-c060-4771-b195-d1c1d4687584","Type":"ContainerStarted","Data":"9ff3b51a1b3b711b16984b1f2e1018c03b433babe1517bf3212ab64d8754665e"} Sep 11 17:55:07.826230 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:07.826178 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" podStartSLOduration=1.9880013380000001 podStartE2EDuration="4.826161047s" totalImagesPullingTime="2.838159709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:07.825951049 +0000 UTC m=+51.857012281" watchObservedRunningTime="2026-09-11 17:55:07.826161047 +0000 UTC m=+51.857222279" Sep 11 17:55:08.136024 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:08.135992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:08.136256 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:08.136149 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 17:55:08.136256 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:08.136223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls podName:522b462e-cb2f-4b6a-9a16-d84758954478 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:24.136205917 +0000 UTC m=+68.167267133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls") pod "dns-default-wlz59" (UID: "522b462e-cb2f-4b6a-9a16-d84758954478") : secret "dns-default-metrics-tls" not found Sep 11 17:55:09.064869 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.064840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:09.147960 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.147928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:09.496261 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.496225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:09.598376 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.598344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:09.649592 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.649558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:09.649740 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.649705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:09.649798 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:09.649708 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 17:55:09.649798 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:09.649783 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:25.649767958 +0000 UTC m=+69.680829183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : configmap references non-existent config key: service-ca.crt Sep 11 17:55:09.649889 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:09.649817 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs podName:66738241-d01b-4e83-9c45-1be1afac1e1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:25.649797345 +0000 UTC m=+69.680858574 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs") pod "router-default-64477965b8-nbp56" (UID: "66738241-d01b-4e83-9c45-1be1afac1e1e") : secret "router-metrics-certs-default" not found Sep 11 17:55:09.772478 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.772388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:09.863938 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:09.863909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:10.498536 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:10.498508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:10.609542 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:10.609487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:10.717630 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:10.717599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:10.813441 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:10.813341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:11.362597 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.362564 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-zdkkf"] Sep 11 17:55:11.364742 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.364718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.367710 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.367690 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wzd9b\"" Sep 11 17:55:11.367916 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.367894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 17:55:11.369011 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.368996 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 17:55:11.369087 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.369053 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 17:55:11.369146 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.369084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 17:55:11.375013 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.374993 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-zdkkf"] Sep 11 17:55:11.428647 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.428616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:11.461594 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.461543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s89x\" (UniqueName: \"kubernetes.io/projected/a49f6234-0792-4c6a-9c43-5d8f112874a3-kube-api-access-5s89x\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.461740 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.461667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a49f6234-0792-4c6a-9c43-5d8f112874a3-signing-key\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.461740 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.461735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a49f6234-0792-4c6a-9c43-5d8f112874a3-signing-cabundle\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.562271 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.562232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a49f6234-0792-4c6a-9c43-5d8f112874a3-signing-cabundle\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.562754 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.562283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s89x\" (UniqueName: \"kubernetes.io/projected/a49f6234-0792-4c6a-9c43-5d8f112874a3-kube-api-access-5s89x\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.562754 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.562359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a49f6234-0792-4c6a-9c43-5d8f112874a3-signing-key\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.563011 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.562991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a49f6234-0792-4c6a-9c43-5d8f112874a3-signing-cabundle\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.564823 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.564806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a49f6234-0792-4c6a-9c43-5d8f112874a3-signing-key\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.572033 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.572010 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s89x\" (UniqueName: \"kubernetes.io/projected/a49f6234-0792-4c6a-9c43-5d8f112874a3-kube-api-access-5s89x\") pod \"service-ca-568869fdc5-zdkkf\" (UID: \"a49f6234-0792-4c6a-9c43-5d8f112874a3\") " pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.673353 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.673300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" Sep 11 17:55:11.941267 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:11.941241 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-zdkkf"] Sep 11 17:55:11.943572 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:11.943545 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49f6234_0792_4c6a_9c43_5d8f112874a3.slice/crio-c374561a5204b1c702331257730c9b72bb26ddecd6174920d2c88e767377c0e6 WatchSource:0}: Error finding container c374561a5204b1c702331257730c9b72bb26ddecd6174920d2c88e767377c0e6: Status 404 returned error can't find the container with id c374561a5204b1c702331257730c9b72bb26ddecd6174920d2c88e767377c0e6 Sep 11 17:55:12.428457 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:12.428412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:12.823279 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:12.823195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" event={"ID":"a49f6234-0792-4c6a-9c43-5d8f112874a3","Type":"ContainerStarted","Data":"357c2d08ad70230a6a351421351f5b9e08e204cbc02884c64f1f433675320217"} Sep 11 17:55:12.823279 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:12.823244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" event={"ID":"a49f6234-0792-4c6a-9c43-5d8f112874a3","Type":"ContainerStarted","Data":"c374561a5204b1c702331257730c9b72bb26ddecd6174920d2c88e767377c0e6"} Sep 11 17:55:12.848375 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:12.848318 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-zdkkf" podStartSLOduration=1.8482940719999998 podStartE2EDuration="1.848294072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:12.847918216 +0000 UTC m=+56.878979464" watchObservedRunningTime="2026-09-11 17:55:12.848294072 +0000 UTC m=+56.879355303" Sep 11 17:55:12.906406 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:12.906376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:13.119893 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.119862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:13.221166 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.221132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:13.521656 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.521549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:13.578564 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.578524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:13.578852 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.578579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:13.579264 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:13.579233 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 17:55:13.579347 ip-10-0-141-255 kubenswrapper[2579]: E0911 17:55:13.579312 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls podName:c0cf0486-0899-4578-abb2-3667f19ad281 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:29.579292399 +0000 UTC m=+73.610353621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5zn7n" (UID: "c0cf0486-0899-4578-abb2-3667f19ad281") : secret "insights-runtime-extractor-tls" not found Sep 11 17:55:13.579347 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.579228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0cf0486-0899-4578-abb2-3667f19ad281-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:13.662608 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.662551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:13.783104 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.783001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:13.849035 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:13.848985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:14.229893 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:14.229807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:14.586854 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:14.586742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:15.190855 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:15.190810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:15.613603 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:15.613507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:15.784292 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:15.784255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:16.025379 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:16.025348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:16.406607 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:16.406568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:16.590269 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:16.590242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:16.695795 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:16.695725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:16.888229 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:16.888197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:17.008078 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:17.007999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:17.110488 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:17.110460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:17.748821 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:17.748789 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j545j" Sep 11 17:55:17.791676 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:17.791633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:17.890874 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:17.890848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:18.195627 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:18.195591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:18.384131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:18.384093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:18.594401 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:18.594287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:18.980526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:18.980488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:19.398558 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:19.398524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:19.998180 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:19.998147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:20.205726 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:20.205695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:20.385726 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:20.385700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:20.787866 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:20.787770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:20.990557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:20.990527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:21.389978 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:21.389944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:22.248122 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.248084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:22.249091 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.249072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:55:22.249173 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.249132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:55:22.249173 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.249171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:55:22.252130 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.252115 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 17:55:22.253665 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.253652 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 17:55:22.253724 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.253655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 17:55:22.262615 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.262592 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 17:55:22.262755 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.262701 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e12fa326-e7dc-47bf-93d9-dd1f0344a2c5-original-pull-secret\") pod \"global-pull-secret-syncer-z99jr\" (UID: \"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5\") " pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:55:22.262755 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.262714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3391cf32-789c-4ed8-80b9-65af2d2a6955-metrics-certs\") pod \"network-metrics-daemon-sscnw\" (UID: \"3391cf32-789c-4ed8-80b9-65af2d2a6955\") " pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:55:22.273527 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.273504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tz92\" (UniqueName: \"kubernetes.io/projected/0383e6d5-6477-44cf-b7a0-3d00c9d55472-kube-api-access-6tz92\") pod \"network-check-target-t8wml\" (UID: \"0383e6d5-6477-44cf-b7a0-3d00c9d55472\") " pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:55:22.550901 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.550823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-64859\"" Sep 11 17:55:22.553301 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.553287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z99jr" Sep 11 17:55:22.558235 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.558213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sscnw" Sep 11 17:55:22.561007 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.560983 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhttf\"" Sep 11 17:55:22.568388 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.568361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:55:22.581371 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.581341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:22.797231 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.797191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:22.829982 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.829858 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sscnw"] Sep 11 17:55:22.830565 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.830547 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z99jr"] Sep 11 17:55:22.832739 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:22.832707 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12fa326_e7dc_47bf_93d9_dd1f0344a2c5.slice/crio-6643516b75378643321f1c39babcd1a2514dcf8c06f084bc58ed96caaa634bbf WatchSource:0}: Error finding container 6643516b75378643321f1c39babcd1a2514dcf8c06f084bc58ed96caaa634bbf: Status 404 returned error can't find the container with id 6643516b75378643321f1c39babcd1a2514dcf8c06f084bc58ed96caaa634bbf Sep 11 17:55:22.833215 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:22.833185 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3391cf32_789c_4ed8_80b9_65af2d2a6955.slice/crio-964257f9a5efcdd0ccc7176c383364cb619d9a58c9534deb7f11a589a100219d WatchSource:0}: Error finding container 964257f9a5efcdd0ccc7176c383364cb619d9a58c9534deb7f11a589a100219d: Status 404 returned error can't find the container with id 964257f9a5efcdd0ccc7176c383364cb619d9a58c9534deb7f11a589a100219d Sep 11 17:55:22.851097 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.851067 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z99jr" event={"ID":"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5","Type":"ContainerStarted","Data":"6643516b75378643321f1c39babcd1a2514dcf8c06f084bc58ed96caaa634bbf"} Sep 11 17:55:22.852229 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.852208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sscnw" event={"ID":"3391cf32-789c-4ed8-80b9-65af2d2a6955","Type":"ContainerStarted","Data":"964257f9a5efcdd0ccc7176c383364cb619d9a58c9534deb7f11a589a100219d"} Sep 11 17:55:22.857196 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:22.857175 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8wml"] Sep 11 17:55:22.860848 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:22.860828 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0383e6d5_6477_44cf_b7a0_3d00c9d55472.slice/crio-159623a4f0639c720747c82d1de435746533a012d0123d9ba7dfc458b57bf899 WatchSource:0}: Error finding container 159623a4f0639c720747c82d1de435746533a012d0123d9ba7dfc458b57bf899: Status 404 returned error can't find the container with id 159623a4f0639c720747c82d1de435746533a012d0123d9ba7dfc458b57bf899 Sep 11 17:55:23.012714 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:23.012683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:23.194198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:23.194169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:23.576525 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:23.576446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:23.856913 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:23.856835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8wml" event={"ID":"0383e6d5-6477-44cf-b7a0-3d00c9d55472","Type":"ContainerStarted","Data":"159623a4f0639c720747c82d1de435746533a012d0123d9ba7dfc458b57bf899"} Sep 11 17:55:24.164999 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.164967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:24.168850 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.168828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/522b462e-cb2f-4b6a-9a16-d84758954478-metrics-tls\") pod \"dns-default-wlz59\" (UID: \"522b462e-cb2f-4b6a-9a16-d84758954478\") " pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:24.310954 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.310896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:24.438812 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.438773 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cj7w5\"" Sep 11 17:55:24.442794 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.442765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:24.721466 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.721355 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wlz59"] Sep 11 17:55:24.726711 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:24.726682 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522b462e_cb2f_4b6a_9a16_d84758954478.slice/crio-6eedc3b4441cf90def50f0a83bbbafc96e4359eb384a4dd98e6bc5e089d87176 WatchSource:0}: Error finding container 6eedc3b4441cf90def50f0a83bbbafc96e4359eb384a4dd98e6bc5e089d87176: Status 404 returned error can't find the container with id 6eedc3b4441cf90def50f0a83bbbafc96e4359eb384a4dd98e6bc5e089d87176 Sep 11 17:55:24.862093 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.862042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wlz59" event={"ID":"522b462e-cb2f-4b6a-9a16-d84758954478","Type":"ContainerStarted","Data":"6eedc3b4441cf90def50f0a83bbbafc96e4359eb384a4dd98e6bc5e089d87176"} Sep 11 17:55:24.864198 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.864174 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sscnw" event={"ID":"3391cf32-789c-4ed8-80b9-65af2d2a6955","Type":"ContainerStarted","Data":"e41656219bec815e972585093b0ef52770a2dd19b678f68dd92e2bad2630bd64"} Sep 11 17:55:24.864336 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:24.864216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sscnw" event={"ID":"3391cf32-789c-4ed8-80b9-65af2d2a6955","Type":"ContainerStarted","Data":"a9ece9bacf8296dbd895ffd89b0ba99ebd12c37e137703c033087e226915257e"} Sep 11 17:55:25.678299 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.678248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:25.678520 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.678367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:25.679190 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.679157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66738241-d01b-4e83-9c45-1be1afac1e1e-service-ca-bundle\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:25.681248 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.681226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66738241-d01b-4e83-9c45-1be1afac1e1e-metrics-certs\") pod \"router-default-64477965b8-nbp56\" (UID: \"66738241-d01b-4e83-9c45-1be1afac1e1e\") " pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:25.879477 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.879438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:25.914483 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.914454 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bj5dv\"" Sep 11 17:55:25.922438 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.922183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:25.995330 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:25.995242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:27.732671 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:27.732556 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sscnw" podStartSLOduration=69.581603054 podStartE2EDuration="1m10.73253079s" totalImagesPullingTime="1.150927736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:24.884322896 +0000 UTC m=+68.915384128" watchObservedRunningTime="2026-09-11 17:55:27.73253079 +0000 UTC m=+71.763592022" Sep 11 17:55:27.733102 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:27.733079 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64477965b8-nbp56"] Sep 11 17:55:28.531673 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.531648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:28.628714 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.628678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:28.731220 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.731188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:28.830819 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.830792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:28.878274 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.878241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64477965b8-nbp56" event={"ID":"66738241-d01b-4e83-9c45-1be1afac1e1e","Type":"ContainerStarted","Data":"3cfdb3718817e386963eee1d6f369a781ab3dfe278a634e3de2d4e75a50accab"} Sep 11 17:55:28.878407 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.878299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64477965b8-nbp56" event={"ID":"66738241-d01b-4e83-9c45-1be1afac1e1e","Type":"ContainerStarted","Data":"ba1050438b89ba47e11e4c3b1ca3cd7c6ee0dbe41ff14df9d895ded6b96982a8"} Sep 11 17:55:28.879903 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.879836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8wml" event={"ID":"0383e6d5-6477-44cf-b7a0-3d00c9d55472","Type":"ContainerStarted","Data":"f01b4a5af604b43f0f2866953f6755fdee472052d42e5a0f4fe89342e9f3e464"} Sep 11 17:55:28.879903 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.879882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:55:28.881651 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.881625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wlz59" event={"ID":"522b462e-cb2f-4b6a-9a16-d84758954478","Type":"ContainerStarted","Data":"86247f59d5e07bfbb642eae93704806ab8ad1353fc9d2f354e4ef183a86c5975"} Sep 11 17:55:28.881731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.881669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wlz59" event={"ID":"522b462e-cb2f-4b6a-9a16-d84758954478","Type":"ContainerStarted","Data":"cfff02e85cf1e647f9135f95f9171b25fef392eb48dd9acfc16a484b5d6ab570"} Sep 11 17:55:28.881800 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.881735 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:28.882972 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.882956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z99jr" event={"ID":"e12fa326-e7dc-47bf-93d9-dd1f0344a2c5","Type":"ContainerStarted","Data":"23240f9c06827c5a0a2d2863fca5553276f9a0bc69d419e0ace390c941accb71"} Sep 11 17:55:28.900241 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.900207 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64477965b8-nbp56" podStartSLOduration=35.900195496 podStartE2EDuration="35.900195496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:28.898908473 +0000 UTC m=+72.929969716" watchObservedRunningTime="2026-09-11 17:55:28.900195496 +0000 UTC m=+72.931256725" Sep 11 17:55:28.916539 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.916489 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z99jr" podStartSLOduration=66.747240374 podStartE2EDuration="1m11.916479231s" totalImagesPullingTime="5.169238857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:28.915471978 +0000 UTC m=+72.946533228" watchObservedRunningTime="2026-09-11 17:55:28.916479231 +0000 UTC m=+72.947540463" Sep 11 17:55:28.923095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.923070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:28.925526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.925506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:28.932122 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.932085 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t8wml" podStartSLOduration=66.79266998 podStartE2EDuration="1m11.932071751s" totalImagesPullingTime="5.139401771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:28.931350193 +0000 UTC m=+72.962411436" watchObservedRunningTime="2026-09-11 17:55:28.932071751 +0000 UTC m=+72.963132981" Sep 11 17:55:28.950529 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:28.950487 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wlz59" podStartSLOduration=33.343397242 podStartE2EDuration="36.950472839s" totalImagesPullingTime="3.607075597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:28.94964308 +0000 UTC m=+72.980704329" watchObservedRunningTime="2026-09-11 17:55:28.950472839 +0000 UTC m=+72.981534069" Sep 11 17:55:29.609575 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:29.609528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:29.612067 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:29.612043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0cf0486-0899-4578-abb2-3667f19ad281-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5zn7n\" (UID: \"c0cf0486-0899-4578-abb2-3667f19ad281\") " pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:29.802171 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:29.802136 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntvnk\"" Sep 11 17:55:29.807799 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:29.807778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5zn7n" Sep 11 17:55:29.890014 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:29.889983 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:29.891526 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:29.891505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64477965b8-nbp56" Sep 11 17:55:30.076274 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:30.076243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5zn7n"] Sep 11 17:55:30.079247 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:30.079222 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cf0486_0899_4578_abb2_3667f19ad281.slice/crio-5393e294ba291950f6e3c7de9316cf38eee680c711f574555a718e4271820cec WatchSource:0}: Error finding container 5393e294ba291950f6e3c7de9316cf38eee680c711f574555a718e4271820cec: Status 404 returned error can't find the container with id 5393e294ba291950f6e3c7de9316cf38eee680c711f574555a718e4271820cec Sep 11 17:55:30.892477 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:30.892443 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zn7n" event={"ID":"c0cf0486-0899-4578-abb2-3667f19ad281","Type":"ContainerStarted","Data":"0ec601ffef1402f823461abf6d25aca51d48b24afc6a507c15f8775b9c4ea4d6"} Sep 11 17:55:30.892795 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:30.892504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zn7n" event={"ID":"c0cf0486-0899-4578-abb2-3667f19ad281","Type":"ContainerStarted","Data":"24a2f9ad9258a2d878132a352d5de5f5835ce9a0dd6942498c382ba40598754c"} Sep 11 17:55:30.892795 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:30.892530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zn7n" event={"ID":"c0cf0486-0899-4578-abb2-3667f19ad281","Type":"ContainerStarted","Data":"5393e294ba291950f6e3c7de9316cf38eee680c711f574555a718e4271820cec"} Sep 11 17:55:32.901487 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:32.901449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5zn7n" event={"ID":"c0cf0486-0899-4578-abb2-3667f19ad281","Type":"ContainerStarted","Data":"7790b5631e0fe6e50a6bde81449dd254005acca5cc1a7a8129b1e67293ffe552"} Sep 11 17:55:34.540563 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.540512 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5zn7n" podStartSLOduration=35.56118943 podStartE2EDuration="37.540492989s" totalImagesPullingTime="1.979303559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:32.935561463 +0000 UTC m=+76.966622705" watchObservedRunningTime="2026-09-11 17:55:34.540492989 +0000 UTC m=+78.571554219" Sep 11 17:55:34.541360 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.541339 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh"] Sep 11 17:55:34.543237 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.543219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.546637 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.546604 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ctwj9\"" Sep 11 17:55:34.546735 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.546679 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 17:55:34.548060 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.548043 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 17:55:34.548157 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.548063 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 17:55:34.548467 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.548448 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 17:55:34.579478 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.579457 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh"] Sep 11 17:55:34.648887 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.648855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/361a1110-4ff6-4a76-9ad1-fd0f3585ce61-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-677f597f46-79bmh\" (UID: \"361a1110-4ff6-4a76-9ad1-fd0f3585ce61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.649024 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.648904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cjs\" (UniqueName: \"kubernetes.io/projected/361a1110-4ff6-4a76-9ad1-fd0f3585ce61-kube-api-access-85cjs\") pod \"managed-serviceaccount-addon-agent-677f597f46-79bmh\" (UID: \"361a1110-4ff6-4a76-9ad1-fd0f3585ce61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.698996 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.698966 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm"] Sep 11 17:55:34.701147 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.701132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:34.704464 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.704440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-btwcg\"" Sep 11 17:55:34.704751 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.704732 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 17:55:34.717941 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.717916 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm"] Sep 11 17:55:34.735979 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.735954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:34.749677 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.749655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/361a1110-4ff6-4a76-9ad1-fd0f3585ce61-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-677f597f46-79bmh\" (UID: \"361a1110-4ff6-4a76-9ad1-fd0f3585ce61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.749790 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.749692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/585b8669-f805-474d-9b75-d97d91055862-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-4xqbm\" (UID: \"585b8669-f805-474d-9b75-d97d91055862\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:34.749790 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.749729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cjs\" (UniqueName: \"kubernetes.io/projected/361a1110-4ff6-4a76-9ad1-fd0f3585ce61-kube-api-access-85cjs\") pod \"managed-serviceaccount-addon-agent-677f597f46-79bmh\" (UID: \"361a1110-4ff6-4a76-9ad1-fd0f3585ce61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.752456 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.752438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/361a1110-4ff6-4a76-9ad1-fd0f3585ce61-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-677f597f46-79bmh\" (UID: \"361a1110-4ff6-4a76-9ad1-fd0f3585ce61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.762045 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.762024 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55b6cdf8db-d84sb"] Sep 11 17:55:34.763971 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.763955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.767341 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.767325 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 17:55:34.767475 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.767325 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5487c\"" Sep 11 17:55:34.767475 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.767460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 17:55:34.767586 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.767549 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 17:55:34.770969 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.770943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cjs\" (UniqueName: \"kubernetes.io/projected/361a1110-4ff6-4a76-9ad1-fd0f3585ce61-kube-api-access-85cjs\") pod \"managed-serviceaccount-addon-agent-677f597f46-79bmh\" (UID: \"361a1110-4ff6-4a76-9ad1-fd0f3585ce61\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.774130 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.774112 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 17:55:34.781660 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.781613 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b6cdf8db-d84sb"] Sep 11 17:55:34.840933 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.840903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:34.850272 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/585b8669-f805-474d-9b75-d97d91055862-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-4xqbm\" (UID: \"585b8669-f805-474d-9b75-d97d91055862\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:34.850392 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpffp\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-kube-api-access-vpffp\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850392 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-registry-certificates\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850392 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-registry-tls\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850392 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-image-registry-private-configuration\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-bound-sa-token\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-ca-trust-extracted\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-trusted-ca\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.850636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.850615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-installation-pull-secrets\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.852935 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.852910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/585b8669-f805-474d-9b75-d97d91055862-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-4xqbm\" (UID: \"585b8669-f805-474d-9b75-d97d91055862\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:34.862787 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.862769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" Sep 11 17:55:34.945847 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.945815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:34.951379 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-image-registry-private-configuration\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-bound-sa-token\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-ca-trust-extracted\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-trusted-ca\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-installation-pull-secrets\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpffp\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-kube-api-access-vpffp\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951752 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-registry-certificates\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951752 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-registry-tls\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.951868 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.951848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-ca-trust-extracted\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.952385 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.952349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-registry-certificates\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.952675 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.952654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-trusted-ca\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.953931 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.953916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-installation-pull-secrets\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.954433 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.954407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-image-registry-private-configuration\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.954573 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.954558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-registry-tls\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.963447 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.963407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-bound-sa-token\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:34.963684 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:34.963668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpffp\" (UniqueName: \"kubernetes.io/projected/253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5-kube-api-access-vpffp\") pod \"image-registry-55b6cdf8db-d84sb\" (UID: \"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5\") " pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:35.009883 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.009844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:35.059131 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.059098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:35.077905 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.077875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:35.138600 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.138573 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh"] Sep 11 17:55:35.142750 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:35.142724 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod361a1110_4ff6_4a76_9ad1_fd0f3585ce61.slice/crio-23537c179f0c99c1c13d33a6321b9896293cd95827fe016c96703b59f81419e2 WatchSource:0}: Error finding container 23537c179f0c99c1c13d33a6321b9896293cd95827fe016c96703b59f81419e2: Status 404 returned error can't find the container with id 23537c179f0c99c1c13d33a6321b9896293cd95827fe016c96703b59f81419e2 Sep 11 17:55:35.278367 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.278344 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm"] Sep 11 17:55:35.280098 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:35.280076 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585b8669_f805_474d_9b75_d97d91055862.slice/crio-84ab8d964503e9f55d027738751a9b6bc5d3e8377f96eb25671e4b707f587d8f WatchSource:0}: Error finding container 84ab8d964503e9f55d027738751a9b6bc5d3e8377f96eb25671e4b707f587d8f: Status 404 returned error can't find the container with id 84ab8d964503e9f55d027738751a9b6bc5d3e8377f96eb25671e4b707f587d8f Sep 11 17:55:35.351491 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.351411 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b6cdf8db-d84sb"] Sep 11 17:55:35.353291 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:35.353263 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253a9bc2_5bea_4a7f_8ff6_dd21410eb2a5.slice/crio-2ec02c1735d6b91f176e9058088d59189e87802e792aec6ab504a8be445c989e WatchSource:0}: Error finding container 2ec02c1735d6b91f176e9058088d59189e87802e792aec6ab504a8be445c989e: Status 404 returned error can't find the container with id 2ec02c1735d6b91f176e9058088d59189e87802e792aec6ab504a8be445c989e Sep 11 17:55:35.915908 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.915872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" event={"ID":"361a1110-4ff6-4a76-9ad1-fd0f3585ce61","Type":"ContainerStarted","Data":"23537c179f0c99c1c13d33a6321b9896293cd95827fe016c96703b59f81419e2"} Sep 11 17:55:35.919311 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.919276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" event={"ID":"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5","Type":"ContainerStarted","Data":"940e215445b4338a7fef15eef528d89966580cc529d50c6605891444d5ad502a"} Sep 11 17:55:35.919446 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.919335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" event={"ID":"253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5","Type":"ContainerStarted","Data":"2ec02c1735d6b91f176e9058088d59189e87802e792aec6ab504a8be445c989e"} Sep 11 17:55:35.919821 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.919793 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:35.922210 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.922183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" event={"ID":"585b8669-f805-474d-9b75-d97d91055862","Type":"ContainerStarted","Data":"84ab8d964503e9f55d027738751a9b6bc5d3e8377f96eb25671e4b707f587d8f"} Sep 11 17:55:35.943873 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:35.943818 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" podStartSLOduration=1.943801704 podStartE2EDuration="1.943801704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:35.942324806 +0000 UTC m=+79.973386070" watchObservedRunningTime="2026-09-11 17:55:35.943801704 +0000 UTC m=+79.974862935" Sep 11 17:55:36.927093 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:36.927046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" event={"ID":"585b8669-f805-474d-9b75-d97d91055862","Type":"ContainerStarted","Data":"dc289a23392f373cfccff866ddc603a4a32351d4c654797f03ea0708ddc16101"} Sep 11 17:55:36.927731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:36.927710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:36.933670 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:36.933648 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" Sep 11 17:55:36.945941 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:36.945893 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-4xqbm" podStartSLOduration=1.656696938 podStartE2EDuration="2.945879s" totalImagesPullingTime="1.289182062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:36.944777018 +0000 UTC m=+80.975838264" watchObservedRunningTime="2026-09-11 17:55:36.945879 +0000 UTC m=+80.976940209" Sep 11 17:55:37.309678 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.309588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:37.431549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.431507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:37.792906 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.792839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:37.880060 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.880025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:37.932348 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.932315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" event={"ID":"361a1110-4ff6-4a76-9ad1-fd0f3585ce61","Type":"ContainerStarted","Data":"c52396d6b1edaba065b0c1f8385dabb2e65de4eb7f65069649fb54008efcb0e7"} Sep 11 17:55:37.951486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.951388 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-677f597f46-79bmh" podStartSLOduration=1.404859737 podStartE2EDuration="3.951374051s" totalImagesPullingTime="2.546514314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:37.95014175 +0000 UTC m=+81.981202981" watchObservedRunningTime="2026-09-11 17:55:37.951374051 +0000 UTC m=+81.982435281" Sep 11 17:55:37.990258 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:37.990234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:38.891407 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:38.891381 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wlz59" Sep 11 17:55:40.551840 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:40.551800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:40.826872 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:40.826793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:40.914798 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:40.914768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:42.917438 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:42.917389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.025857 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.025825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.192050 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.191965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.267247 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.267211 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m"] Sep 11 17:55:43.272436 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.272391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.275745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.275722 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 17:55:43.276061 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.276043 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 17:55:43.276851 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.276833 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 17:55:43.277396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.277377 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 17:55:43.277511 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.277429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 17:55:43.277511 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.277429 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 17:55:43.277644 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.277377 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4f9vx\"" Sep 11 17:55:43.310331 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.310293 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ckmm"] Sep 11 17:55:43.314124 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.314099 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m"] Sep 11 17:55:43.314256 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.314235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.318455 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.318437 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 17:55:43.319229 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.319084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 17:55:43.319326 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.319298 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97vtp\"" Sep 11 17:55:43.320050 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.320035 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 17:55:43.349351 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.349329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.411205 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe9495a1-b461-401a-bb7e-8c177b56c9bf-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.411205 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdsq\" (UniqueName: \"kubernetes.io/projected/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-api-access-hjdsq\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-metrics-client-ca\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-textfile\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-tls\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-wtmp\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-root\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.411731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe9495a1-b461-401a-bb7e-8c177b56c9bf-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.411731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-sys\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.411731 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.411602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69wg\" (UniqueName: \"kubernetes.io/projected/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-kube-api-access-j69wg\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.512955 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.512867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.512955 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.512898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-textfile\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.512955 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.512915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-tls\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.512955 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.512942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-wtmp\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.512990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-root\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe9495a1-b461-401a-bb7e-8c177b56c9bf-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-root\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-sys\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69wg\" (UniqueName: \"kubernetes.io/projected/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-kube-api-access-j69wg\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe9495a1-b461-401a-bb7e-8c177b56c9bf-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513214 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-wtmp\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdsq\" (UniqueName: \"kubernetes.io/projected/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-api-access-hjdsq\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513277 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-metrics-client-ca\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513900 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-textfile\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.513900 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513900 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513900 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fe9495a1-b461-401a-bb7e-8c177b56c9bf-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.513900 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.513466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-sys\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.514197 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.514165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe9495a1-b461-401a-bb7e-8c177b56c9bf-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.514197 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.514202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.514648 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.514617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-metrics-client-ca\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.515332 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.514946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.516240 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.516216 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.516409 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.516389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.516479 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.516445 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.516808 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.516792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-node-exporter-tls\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.522567 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.522547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69wg\" (UniqueName: \"kubernetes.io/projected/e33cb2f7-0e24-4b5e-b44a-d85c5e018e48-kube-api-access-j69wg\") pod \"node-exporter-6ckmm\" (UID: \"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48\") " pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.522663 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.522625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdsq\" (UniqueName: \"kubernetes.io/projected/fe9495a1-b461-401a-bb7e-8c177b56c9bf-kube-api-access-hjdsq\") pod \"kube-state-metrics-59fbdd77d-84p4m\" (UID: \"fe9495a1-b461-401a-bb7e-8c177b56c9bf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.581017 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.580986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.581869 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.581851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" Sep 11 17:55:43.624636 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.624606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ckmm" Sep 11 17:55:43.638341 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:43.638311 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33cb2f7_0e24_4b5e_b44a_d85c5e018e48.slice/crio-13dc513e9413f4e9a6c2a4cb435c70b9f7665cacd66e95423334397d3281ca8d WatchSource:0}: Error finding container 13dc513e9413f4e9a6c2a4cb435c70b9f7665cacd66e95423334397d3281ca8d: Status 404 returned error can't find the container with id 13dc513e9413f4e9a6c2a4cb435c70b9f7665cacd66e95423334397d3281ca8d Sep 11 17:55:43.682270 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.682236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.782759 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.782671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:43.856482 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.856450 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m"] Sep 11 17:55:43.860400 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:43.860378 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9495a1_b461_401a_bb7e_8c177b56c9bf.slice/crio-14b4b80028d25e1f614d2a552a5b5c23836b0e8f1b1b4447edfc6ed72a774023 WatchSource:0}: Error finding container 14b4b80028d25e1f614d2a552a5b5c23836b0e8f1b1b4447edfc6ed72a774023: Status 404 returned error can't find the container with id 14b4b80028d25e1f614d2a552a5b5c23836b0e8f1b1b4447edfc6ed72a774023 Sep 11 17:55:43.958056 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.958025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" event={"ID":"fe9495a1-b461-401a-bb7e-8c177b56c9bf","Type":"ContainerStarted","Data":"14b4b80028d25e1f614d2a552a5b5c23836b0e8f1b1b4447edfc6ed72a774023"} Sep 11 17:55:43.959248 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.959215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ckmm" event={"ID":"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48","Type":"ContainerStarted","Data":"13dc513e9413f4e9a6c2a4cb435c70b9f7665cacd66e95423334397d3281ca8d"} Sep 11 17:55:43.974542 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:43.974519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:44.163216 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:44.163171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:44.244045 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:44.244013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:44.920012 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:44.919965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:44.964718 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:44.964680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ckmm" event={"ID":"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48","Type":"ContainerStarted","Data":"b51c3459bb03a5f65eb313daa040e9794528467c103237582228881fc5868859"} Sep 11 17:55:45.970295 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:45.970254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" event={"ID":"fe9495a1-b461-401a-bb7e-8c177b56c9bf","Type":"ContainerStarted","Data":"6f2f21ae1277e06e3914a20d9d48acba8f8bfe314b09859600415d4e3c2c9581"} Sep 11 17:55:45.970737 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:45.970324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" event={"ID":"fe9495a1-b461-401a-bb7e-8c177b56c9bf","Type":"ContainerStarted","Data":"e11c373b502f43f925870d9ae43243e70c1b27170a36451a0fe797bf7468059e"} Sep 11 17:55:45.970737 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:45.970349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" event={"ID":"fe9495a1-b461-401a-bb7e-8c177b56c9bf","Type":"ContainerStarted","Data":"aabf9ea6585feeff506889a7247ac66923a903b339b30745edb5d22c439181ec"} Sep 11 17:55:45.971783 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:45.971762 2579 generic.go:356] "Generic (PLEG): container finished" podID="e33cb2f7-0e24-4b5e-b44a-d85c5e018e48" containerID="b51c3459bb03a5f65eb313daa040e9794528467c103237582228881fc5868859" exitCode=0 Sep 11 17:55:45.971880 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:45.971790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ckmm" event={"ID":"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48","Type":"ContainerDied","Data":"b51c3459bb03a5f65eb313daa040e9794528467c103237582228881fc5868859"} Sep 11 17:55:45.994457 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:45.994392 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-84p4m" podStartSLOduration=1.753167197 podStartE2EDuration="2.994375307s" totalImagesPullingTime="1.24120811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:45.99291668 +0000 UTC m=+90.023977932" watchObservedRunningTime="2026-09-11 17:55:45.994375307 +0000 UTC m=+90.025436538" Sep 11 17:55:46.304249 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.304160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:46.324079 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.324045 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8499f76744-l7s99"] Sep 11 17:55:46.328098 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.328077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.331141 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331122 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nc7f6\"" Sep 11 17:55:46.331461 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331442 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 17:55:46.331577 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331535 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 17:55:46.331637 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331581 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2bp3qvjv1o0mb\"" Sep 11 17:55:46.331834 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331818 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 17:55:46.331900 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331844 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 17:55:46.331977 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.331962 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 17:55:46.346760 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.346738 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8499f76744-l7s99"] Sep 11 17:55:46.435570 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.435537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:46.438763 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.438734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.438894 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.438773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-grpc-tls\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.438894 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.438828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd9d268b-fc51-4b1c-9214-beed628132a3-metrics-client-ca\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.438894 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.438869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.439075 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.438896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.439075 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.438984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvpdq\" (UniqueName: \"kubernetes.io/projected/dd9d268b-fc51-4b1c-9214-beed628132a3-kube-api-access-lvpdq\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.439075 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.439030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.439075 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.439071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-tls\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540179 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-grpc-tls\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd9d268b-fc51-4b1c-9214-beed628132a3-metrics-client-ca\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvpdq\" (UniqueName: \"kubernetes.io/projected/dd9d268b-fc51-4b1c-9214-beed628132a3-kube-api-access-lvpdq\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-tls\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.540616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.540410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.541039 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.541004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd9d268b-fc51-4b1c-9214-beed628132a3-metrics-client-ca\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.543466 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.543441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.543783 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.543761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.543950 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.543929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.543988 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.543972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.544083 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.544068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-grpc-tls\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.544151 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.544130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd9d268b-fc51-4b1c-9214-beed628132a3-secret-thanos-querier-tls\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.557484 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.557376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvpdq\" (UniqueName: \"kubernetes.io/projected/dd9d268b-fc51-4b1c-9214-beed628132a3-kube-api-access-lvpdq\") pod \"thanos-querier-8499f76744-l7s99\" (UID: \"dd9d268b-fc51-4b1c-9214-beed628132a3\") " pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.619779 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.619753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:46.640776 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.640749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:46.763035 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.763006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:46.859357 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.859274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:46.954694 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.954669 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8499f76744-l7s99"] Sep 11 17:55:46.956457 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:46.956411 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9d268b_fc51_4b1c_9214_beed628132a3.slice/crio-1158ca1ea9df0fa1b810322beecb531232c6d27c4bb7354769624b45cfeb9c9b WatchSource:0}: Error finding container 1158ca1ea9df0fa1b810322beecb531232c6d27c4bb7354769624b45cfeb9c9b: Status 404 returned error can't find the container with id 1158ca1ea9df0fa1b810322beecb531232c6d27c4bb7354769624b45cfeb9c9b Sep 11 17:55:46.977547 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.977517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ckmm" event={"ID":"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48","Type":"ContainerStarted","Data":"40c4abe388b4e5f96208ea8622d1e97a413b985c90acdd8c71d282e2991aba99"} Sep 11 17:55:46.977926 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.977571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ckmm" event={"ID":"e33cb2f7-0e24-4b5e-b44a-d85c5e018e48","Type":"ContainerStarted","Data":"54449a4886e76a5b1d5484effeb7de01bc2f202828113211eef3aa92470da2ce"} Sep 11 17:55:46.978849 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:46.978827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"1158ca1ea9df0fa1b810322beecb531232c6d27c4bb7354769624b45cfeb9c9b"} Sep 11 17:55:47.023978 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.023929 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ckmm" podStartSLOduration=2.997078569 podStartE2EDuration="4.023911643s" totalImagesPullingTime="951.193168ms" totalInitContainerRuntime="75.639906ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:47.022319503 +0000 UTC m=+91.053380752" watchObservedRunningTime="2026-09-11 17:55:47.023911643 +0000 UTC m=+91.054973007" Sep 11 17:55:47.048459 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.048433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:47.240034 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.240003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:47.432182 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.432152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:47.661621 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.661588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:47.826967 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.826939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:47.990634 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:47.990562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:48.163965 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.163925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:48.295846 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.295773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:48.391237 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.391199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:48.646625 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.646584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:48.755144 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.755110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:48.990019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.989906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"d256d7c667165d0e51bc7c81a596a6990c28a3493c25b560e4273e9efd503afe"} Sep 11 17:55:48.990019 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:48.989987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"ac2584581e9280242e0ed352da852ffec60e117244824a03e98d64f98c523f49"} Sep 11 17:55:49.257227 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.257140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:49.354837 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.354812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:49.637250 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.637220 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:55:49.640838 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.640819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.644107 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.644084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 17:55:49.644226 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.644086 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 17:55:49.644322 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.644310 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 17:55:49.644355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.644325 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 17:55:49.646103 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.645945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 17:55:49.646103 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.645985 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 17:55:49.646103 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.646090 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 17:55:49.646362 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.646101 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mbw79\"" Sep 11 17:55:49.647562 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.647275 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 17:55:49.649929 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.649906 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-24so8h2m673vk\"" Sep 11 17:55:49.651985 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.650930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 17:55:49.651985 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.651070 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 17:55:49.651985 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.651093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 17:55:49.652177 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.652070 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 17:55:49.652257 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.652238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 17:55:49.654126 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.654108 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 17:55:49.656193 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.656177 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 17:55:49.663783 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.663763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:55:49.695925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.695894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:49.771901 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.771872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.771901 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.771905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772088 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.771928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772088 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.771942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772088 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.771996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772088 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772088 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-config\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfxd\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-kube-api-access-brfxd\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.772486 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.772453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.864384 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.864344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:49.873808 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.873762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.873808 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.873799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874012 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.873844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfxd\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-kube-api-access-brfxd\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874012 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.873885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874012 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.873940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874012 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.873967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874012 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874462 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874566 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874774 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874774 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874774 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874774 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874774 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.874774 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-config\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.875325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.875325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.875325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.875325 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.874949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.875557 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.875393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.877272 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.877244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.877367 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.877257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.877458 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.877398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.877709 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.877585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.878116 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.878089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.878214 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.878189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.878341 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.878296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.878843 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.878816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.878988 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.878933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.879264 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.879239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.879394 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.879373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.879819 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.879799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.880245 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.880088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.880599 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.880582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.880688 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.880663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-config\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.881206 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.881184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.883495 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.883474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfxd\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-kube-api-access-brfxd\") pod \"prometheus-k8s-0\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.953451 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.953410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:55:49.997254 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:49.997217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"85abc69f008c1d8c7a801718c2402ea9f3520cce694152f2bafa0c1fce84d986"} Sep 11 17:55:50.236928 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:50.236906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:55:50.239090 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:55:50.239066 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fd203e_342a_43b2_85ac_028149617994.slice/crio-8514059a1cb724a91eebe18d850fa3355fa960467fab97598338ce2b03e78b8a WatchSource:0}: Error finding container 8514059a1cb724a91eebe18d850fa3355fa960467fab97598338ce2b03e78b8a: Status 404 returned error can't find the container with id 8514059a1cb724a91eebe18d850fa3355fa960467fab97598338ce2b03e78b8a Sep 11 17:55:51.002889 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:51.002847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"8514059a1cb724a91eebe18d850fa3355fa960467fab97598338ce2b03e78b8a"} Sep 11 17:55:51.006789 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:51.006759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"5bdb5ce1d5547976cb2c5204492b5b9da62572dd1865d90b19137bfba751cdce"} Sep 11 17:55:51.006964 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:51.006841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"484c98ebed934cdd0351a7bd4c6f88b5eea501f1bdcf724de6e5e155f6330aea"} Sep 11 17:55:51.006964 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:51.006873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" event={"ID":"dd9d268b-fc51-4b1c-9214-beed628132a3","Type":"ContainerStarted","Data":"dc00624f27dd14940c9fbab3f57d557da6abf693e62481550a1a34aab21a2ad3"} Sep 11 17:55:51.007080 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:51.006980 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:51.031681 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:51.031449 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" podStartSLOduration=2.232665187 podStartE2EDuration="5.031409514s" totalImagesPullingTime="2.798744327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:51.030021729 +0000 UTC m=+95.061082983" watchObservedRunningTime="2026-09-11 17:55:51.031409514 +0000 UTC m=+95.062470744" Sep 11 17:55:52.013070 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:52.013028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:52.014941 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:52.014909 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="22c6266f830d891ee5033f602b37815f2370ab712eb71012482266516f3276dc" exitCode=0 Sep 11 17:55:52.015079 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:52.014989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"22c6266f830d891ee5033f602b37815f2370ab712eb71012482266516f3276dc"} Sep 11 17:55:52.163794 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:52.163750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:55.818164 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:55.818079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:55.919916 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:55.919881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:56.037128 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.037092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"fed5a889bf0fda6634c2da63034d77b1b7974621c961835e0e4f09d6b2356b0d"} Sep 11 17:55:56.037284 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.037158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"4f0700ca72f80d62df103401b63c52fce754f3e5765f734b76431728f8fdf33a"} Sep 11 17:55:56.037284 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.037197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"85ff76737a888a5b7b8cb9b8a236d251f9a6311417ac086b9e489a3966b1ed71"} Sep 11 17:55:56.037284 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.037225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"a1ca09739bd72c557854583b886008fb33c82fd22b818ca12c1315fad6b333b7"} Sep 11 17:55:56.037284 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.037252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"3f360a07d9f55e9f26d92285b9c837be02a9732723cb2a29dcb1f2a3342ea1ec"} Sep 11 17:55:56.037436 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.037289 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerStarted","Data":"20a6c76da429df16bc2c12edfb0273abbeed64a5aa97e74ac5e972c4d55692de"} Sep 11 17:55:56.081078 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.080950 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.132177803 podStartE2EDuration="7.080933216s" totalImagesPullingTime="3.931069491s" totalInitContainerRuntime="17.685922ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:56.080156251 +0000 UTC m=+100.111217500" watchObservedRunningTime="2026-09-11 17:55:56.080933216 +0000 UTC m=+100.111994445" Sep 11 17:55:56.932000 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:56.931969 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55b6cdf8db-d84sb" Sep 11 17:55:57.021362 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:57.021339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8499f76744-l7s99" Sep 11 17:55:57.085307 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:57.085277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:57.189457 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:57.189342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:57.289753 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:57.289721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:57.390166 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:57.390138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:59.145732 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:59.145679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:59.229501 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:59.229469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:59.432384 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:59.432304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:59.533377 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:59.533343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:55:59.891697 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:59.891674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t8wml" Sep 11 17:55:59.954524 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:55:59.954491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:56:02.634390 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:02.634356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:02.736655 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:02.736619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:04.242360 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:04.242297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:04.344199 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:04.344167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:06.374234 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:06.374202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:06.511825 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:06.511790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:08.360310 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:08.360279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:08.453738 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:08.453705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:10.833819 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:10.833790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:10.918372 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:10.918338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:13.123142 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:13.123096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:13.197014 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:13.196982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:13.582801 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:13.582718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:13.685152 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:13.685108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:13.761218 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:13.761189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:14.247830 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:14.247790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:15.533503 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:15.533473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:15.613784 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:15.613748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:16.212611 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:16.212575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:16.313721 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:16.313671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:18.099845 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:18.099818 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64477965b8-nbp56_66738241-d01b-4e83-9c45-1be1afac1e1e/router/0.log" Sep 11 17:56:18.107346 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:18.107327 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9pnbx_4b8b095b-591c-4f3d-97c6-bc7b5833ab38/serve-healthcheck-canary/0.log" Sep 11 17:56:18.251413 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:18.251353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:18.351613 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:18.351525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:18.430935 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:18.430906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:24.145037 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:24.145003 2579 generic.go:356] "Generic (PLEG): container finished" podID="4ace9b19-c060-4771-b195-d1c1d4687584" containerID="9ff3b51a1b3b711b16984b1f2e1018c03b433babe1517bf3212ab64d8754665e" exitCode=0 Sep 11 17:56:24.145468 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:24.145076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" event={"ID":"4ace9b19-c060-4771-b195-d1c1d4687584","Type":"ContainerDied","Data":"9ff3b51a1b3b711b16984b1f2e1018c03b433babe1517bf3212ab64d8754665e"} Sep 11 17:56:24.145468 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:24.145369 2579 scope.go:117] "RemoveContainer" containerID="9ff3b51a1b3b711b16984b1f2e1018c03b433babe1517bf3212ab64d8754665e" Sep 11 17:56:25.150497 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:25.150463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kxr2r" event={"ID":"4ace9b19-c060-4771-b195-d1c1d4687584","Type":"ContainerStarted","Data":"01993e981b90579572c6d6a7eea14a2073f0cc0991a42d56f67d67eb48408a4e"} Sep 11 17:56:40.819699 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:40.819661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:40.896549 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:40.896512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:56:49.954114 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:49.954082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:56:49.970138 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:49.970115 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:56:50.268360 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:56:50.268288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:00.322132 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:00.322098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:02.660109 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:02.660069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:02.759118 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:02.759078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:08.067750 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.067676 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:57:08.068120 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.068009 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="prometheus" containerID="cri-o://20a6c76da429df16bc2c12edfb0273abbeed64a5aa97e74ac5e972c4d55692de" gracePeriod=600 Sep 11 17:57:08.068120 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.068017 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fed5a889bf0fda6634c2da63034d77b1b7974621c961835e0e4f09d6b2356b0d" gracePeriod=600 Sep 11 17:57:08.068120 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.068036 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="thanos-sidecar" containerID="cri-o://a1ca09739bd72c557854583b886008fb33c82fd22b818ca12c1315fad6b333b7" gracePeriod=600 Sep 11 17:57:08.068120 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.068070 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-web" containerID="cri-o://85ff76737a888a5b7b8cb9b8a236d251f9a6311417ac086b9e489a3966b1ed71" gracePeriod=600 Sep 11 17:57:08.068469 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.068070 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="config-reloader" containerID="cri-o://3f360a07d9f55e9f26d92285b9c837be02a9732723cb2a29dcb1f2a3342ea1ec" gracePeriod=600 Sep 11 17:57:08.068469 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.068066 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy" containerID="cri-o://4f0700ca72f80d62df103401b63c52fce754f3e5765f734b76431728f8fdf33a" gracePeriod=600 Sep 11 17:57:08.323095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323032 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="fed5a889bf0fda6634c2da63034d77b1b7974621c961835e0e4f09d6b2356b0d" exitCode=0 Sep 11 17:57:08.323095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323055 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="4f0700ca72f80d62df103401b63c52fce754f3e5765f734b76431728f8fdf33a" exitCode=0 Sep 11 17:57:08.323095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323063 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="85ff76737a888a5b7b8cb9b8a236d251f9a6311417ac086b9e489a3966b1ed71" exitCode=0 Sep 11 17:57:08.323095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323068 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="a1ca09739bd72c557854583b886008fb33c82fd22b818ca12c1315fad6b333b7" exitCode=0 Sep 11 17:57:08.323095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323074 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="3f360a07d9f55e9f26d92285b9c837be02a9732723cb2a29dcb1f2a3342ea1ec" exitCode=0 Sep 11 17:57:08.323095 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323082 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9fd203e-342a-43b2-85ac-028149617994" containerID="20a6c76da429df16bc2c12edfb0273abbeed64a5aa97e74ac5e972c4d55692de" exitCode=0 Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"fed5a889bf0fda6634c2da63034d77b1b7974621c961835e0e4f09d6b2356b0d"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"4f0700ca72f80d62df103401b63c52fce754f3e5765f734b76431728f8fdf33a"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"85ff76737a888a5b7b8cb9b8a236d251f9a6311417ac086b9e489a3966b1ed71"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"a1ca09739bd72c557854583b886008fb33c82fd22b818ca12c1315fad6b333b7"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"3f360a07d9f55e9f26d92285b9c837be02a9732723cb2a29dcb1f2a3342ea1ec"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"20a6c76da429df16bc2c12edfb0273abbeed64a5aa97e74ac5e972c4d55692de"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9fd203e-342a-43b2-85ac-028149617994","Type":"ContainerDied","Data":"8514059a1cb724a91eebe18d850fa3355fa960467fab97598338ce2b03e78b8a"} Sep 11 17:57:08.323396 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.323307 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8514059a1cb724a91eebe18d850fa3355fa960467fab97598338ce2b03e78b8a" Sep 11 17:57:08.324227 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.324210 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:08.463024 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.462993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-0\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463024 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463028 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-config-out\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463259 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463050 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-thanos-prometheus-http-client-file\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463259 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463073 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-metrics-client-certs\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463259 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463259 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-kubelet-serving-ca-bundle\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-metrics-client-ca\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brfxd\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-kube-api-access-brfxd\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463362 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-tls-assets\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463468 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-1\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-2\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-config\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-grpc-tls\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-trusted-ca-bundle\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463635 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-tls\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463662 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-kube-rbac-proxy\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463688 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-web-config\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463715 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-serving-certs-ca-bundle\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.463859 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463741 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-db\") pod \"f9fd203e-342a-43b2-85ac-028149617994\" (UID: \"f9fd203e-342a-43b2-85ac-028149617994\") " Sep 11 17:57:08.464230 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.463915 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.464230 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.464018 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.465371 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.464845 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.465371 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.465010 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.465371 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.465230 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.465371 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.465363 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.466269 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.465990 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.466372 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.466283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:57:08.466464 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.466368 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-config-out" (OuterVolumeSpecName: "config-out") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 17:57:08.466464 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.466440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.466590 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.466498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.466590 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.466560 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.467284 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.467254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 17:57:08.467468 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.467341 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.467876 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.467843 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-config" (OuterVolumeSpecName: "config") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.467876 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.467863 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-kube-api-access-brfxd" (OuterVolumeSpecName: "kube-api-access-brfxd") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "kube-api-access-brfxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:57:08.468068 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.468045 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.468959 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.468939 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.469015 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.468963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.469193 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.469178 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 17:57:08.479660 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.479626 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-web-config" (OuterVolumeSpecName: "web-config") pod "f9fd203e-342a-43b2-85ac-028149617994" (UID: "f9fd203e-342a-43b2-85ac-028149617994"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 17:57:08.564449 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564390 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564449 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564444 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564458 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-kube-rbac-proxy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564471 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-web-config\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564481 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564489 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-db\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564498 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564506 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9fd203e-342a-43b2-85ac-028149617994-config-out\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564514 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564522 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-metrics-client-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564530 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564539 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-configmap-metrics-client-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564554 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfxd\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-kube-api-access-brfxd\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564566 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564574 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9fd203e-342a-43b2-85ac-028149617994-tls-assets\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564582 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564591 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9fd203e-342a-43b2-85ac-028149617994-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564599 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-config\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:08.564616 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:08.564608 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9fd203e-342a-43b2-85ac-028149617994-secret-grpc-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 17:57:09.327320 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.327291 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.352437 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.352395 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:57:09.359820 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.359794 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:57:09.391631 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391600 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:57:09.391866 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391856 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="prometheus" Sep 11 17:57:09.391908 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391869 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="prometheus" Sep 11 17:57:09.391908 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391878 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="thanos-sidecar" Sep 11 17:57:09.391908 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391884 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="thanos-sidecar" Sep 11 17:57:09.391908 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391893 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-web" Sep 11 17:57:09.391908 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391899 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-web" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391914 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391919 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391925 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-thanos" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391931 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-thanos" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391938 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="init-config-reloader" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391945 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="init-config-reloader" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391950 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="config-reloader" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391956 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="config-reloader" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.391994 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.392002 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="config-reloader" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.392008 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-thanos" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.392014 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="thanos-sidecar" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.392018 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="kube-rbac-proxy-web" Sep 11 17:57:09.392048 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.392024 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9fd203e-342a-43b2-85ac-028149617994" containerName="prometheus" Sep 11 17:57:09.396248 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.396228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.399522 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 17:57:09.399637 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 17:57:09.399637 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 17:57:09.399637 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399552 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 17:57:09.399637 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399600 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mbw79\"" Sep 11 17:57:09.399840 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399642 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 17:57:09.399840 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399750 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 17:57:09.399957 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.399941 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 17:57:09.400053 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400039 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 17:57:09.400103 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400093 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 17:57:09.400339 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400323 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 17:57:09.400473 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400454 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 17:57:09.400573 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400474 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-24so8h2m673vk\"" Sep 11 17:57:09.400681 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400665 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 17:57:09.401004 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.400987 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 17:57:09.403806 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.403790 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 17:57:09.405640 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.405624 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 17:57:09.416896 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.416875 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:57:09.470761 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470761 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470962 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470962 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470962 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85qv\" (UniqueName: \"kubernetes.io/projected/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-kube-api-access-l85qv\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470962 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470962 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-config\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.470962 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.470985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471135 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.471355 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.471207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572318 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572518 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572518 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572518 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572518 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572518 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572746 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572746 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572746 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572746 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85qv\" (UniqueName: \"kubernetes.io/projected/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-kube-api-access-l85qv\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.572925 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-config\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.573211 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.573211 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.573211 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.572996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.573211 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.573023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.573586 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.573547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.573703 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.573645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.575165 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.574127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.575165 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.574490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.575835 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.575800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.576126 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.576101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.576195 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.576105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.576195 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.576152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.576477 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.576449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-config\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.576563 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.576480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.576704 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.576678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.577144 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.577080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.577463 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.577442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.577948 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.577924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.578668 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.578642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.578895 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.578872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.578999 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.578982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.578999 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.578991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.579109 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.579095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.583613 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.583598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85qv\" (UniqueName: \"kubernetes.io/projected/cd7e049c-e58d-4b14-bf99-ca4f0148fc47-kube-api-access-l85qv\") pod \"prometheus-k8s-0\" (UID: \"cd7e049c-e58d-4b14-bf99-ca4f0148fc47\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.706448 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.706399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:09.987326 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:09.987081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 17:57:09.990071 ip-10-0-141-255 kubenswrapper[2579]: W0911 17:57:09.990046 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7e049c_e58d_4b14_bf99_ca4f0148fc47.slice/crio-e1317a4367a110a7a2d00b65f99c8fcc057f061c9374132eb1ccef5c51bac5d9 WatchSource:0}: Error finding container e1317a4367a110a7a2d00b65f99c8fcc057f061c9374132eb1ccef5c51bac5d9: Status 404 returned error can't find the container with id e1317a4367a110a7a2d00b65f99c8fcc057f061c9374132eb1ccef5c51bac5d9 Sep 11 17:57:10.333212 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:10.333117 2579 generic.go:356] "Generic (PLEG): container finished" podID="cd7e049c-e58d-4b14-bf99-ca4f0148fc47" containerID="d0f43ebf0e7f36e07ce8054f4e2e05f261d1bdb364d890ab684ebfeb6093a4d9" exitCode=0 Sep 11 17:57:10.333653 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:10.333203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerDied","Data":"d0f43ebf0e7f36e07ce8054f4e2e05f261d1bdb364d890ab684ebfeb6093a4d9"} Sep 11 17:57:10.333653 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:10.333277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"e1317a4367a110a7a2d00b65f99c8fcc057f061c9374132eb1ccef5c51bac5d9"} Sep 11 17:57:10.444376 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:10.444336 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9fd203e-342a-43b2-85ac-028149617994" path="/var/lib/kubelet/pods/f9fd203e-342a-43b2-85ac-028149617994/volumes" Sep 11 17:57:10.848206 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:10.848118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:10.948866 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:10.948825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:11.343560 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.343526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"71be914e08636cac7a863dbc3dcdbbef2e3fd3146060dacf3272aa334b6a2364"} Sep 11 17:57:11.343921 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.343591 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"33f82e8df7d87aa6ca6bc5840e024af3383cc730d6b90b94d747e8f3c3e04ae5"} Sep 11 17:57:11.343921 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.343620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"b0bd2f6d47db83e40f5d9b3e5676fa08005b5376e00edeeea5720dce194f6638"} Sep 11 17:57:11.343921 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.343647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"a049d9bfcc381b53bba02f1e7c9fb9f4c500ceba2db52895bd006aee29fdeb79"} Sep 11 17:57:11.343921 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.343668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"7fc7744b792fbe8543504c722e40e9b3026423c4804297b927b462409455b8f6"} Sep 11 17:57:11.343921 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.343693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd7e049c-e58d-4b14-bf99-ca4f0148fc47","Type":"ContainerStarted","Data":"574d780d31d96e549c33929cb61ac0ada92b4ccc7b685f592bba6ca62504d341"} Sep 11 17:57:11.379651 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:11.379592 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3590851600000002 podStartE2EDuration="2.379571151s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.485991ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:57:11.377013869 +0000 UTC m=+175.408075099" watchObservedRunningTime="2026-09-11 17:57:11.379571151 +0000 UTC m=+175.410632381" Sep 11 17:57:14.706877 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:14.706832 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:57:21.918351 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:21.918313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:22.024096 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:22.024067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:23.865839 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:23.865791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:43.510923 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:43.510894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:48.279602 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:48.279563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:48.347149 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:48.347112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:57:57.132940 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:57:57.132907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:09.707503 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:09.707456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:58:09.723573 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:09.723544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:58:10.588126 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:10.588100 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 17:58:18.053696 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:18.053661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:18.148578 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:18.148543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:29.996856 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:29.996816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:30.117246 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:30.117215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:40.943745 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:40.943650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:41.099650 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:41.099618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:47.003398 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:47.003367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:47.166700 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:47.166663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:57.580599 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:57.580565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:58.307771 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:58.307731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:58.407563 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:58.407530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:58:58.491192 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:58:58.491158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:01.627157 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:01.627115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:01.726018 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:01.725987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:16.368282 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:16.368254 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 17:59:23.487679 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:23.487653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:25.487882 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:25.487832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:27.280811 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:27.280779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:27.394517 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:27.394485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:29.414599 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:29.414566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:29.543288 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:29.543247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:31.979380 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:31.979344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:32.197766 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:32.197734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:34.251366 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:34.251332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:34.355713 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:34.355679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:38.653501 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:38.653461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:38.755304 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:38.755274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:38.988373 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:38.988290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:40.355840 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:40.355815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:40.455023 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:40.454992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:57.797056 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:57.797025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:57.898713 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:57.898674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 17:59:59.997855 ip-10-0-141-255 kubenswrapper[2579]: I0911 17:59:59.997825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:10.329980 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.329889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:10.417584 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.417549 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj"] Sep 11 18:00:10.420652 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.420631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.425472 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.425452 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 18:00:10.427024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.427001 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 18:00:10.427120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.427073 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 18:00:10.427120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.427083 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 18:00:10.427238 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.427088 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wjldh\"" Sep 11 18:00:10.427238 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.427171 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 18:00:10.430596 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.430569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:10.443747 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.443721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj"] Sep 11 18:00:10.514361 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.514326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tczm\" (UniqueName: \"kubernetes.io/projected/b6329017-16da-4de4-aee0-14082b9f0a0f-kube-api-access-9tczm\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.514565 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.514443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6329017-16da-4de4-aee0-14082b9f0a0f-manager-config\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.514565 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.514483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6329017-16da-4de4-aee0-14082b9f0a0f-metrics-cert\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.514565 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.514539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6329017-16da-4de4-aee0-14082b9f0a0f-cert\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.615159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.615072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6329017-16da-4de4-aee0-14082b9f0a0f-manager-config\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.615159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.615111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6329017-16da-4de4-aee0-14082b9f0a0f-metrics-cert\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.615159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.615136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6329017-16da-4de4-aee0-14082b9f0a0f-cert\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.615442 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.615179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tczm\" (UniqueName: \"kubernetes.io/projected/b6329017-16da-4de4-aee0-14082b9f0a0f-kube-api-access-9tczm\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.615754 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.615724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6329017-16da-4de4-aee0-14082b9f0a0f-manager-config\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.617699 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.617677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6329017-16da-4de4-aee0-14082b9f0a0f-metrics-cert\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.617784 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.617722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6329017-16da-4de4-aee0-14082b9f0a0f-cert\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.624846 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.624817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tczm\" (UniqueName: \"kubernetes.io/projected/b6329017-16da-4de4-aee0-14082b9f0a0f-kube-api-access-9tczm\") pod \"lws-controller-manager-ffbd766d4-jc9pj\" (UID: \"b6329017-16da-4de4-aee0-14082b9f0a0f\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:10.729384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:10.729342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:11.030157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:11.030135 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj"] Sep 11 18:00:11.031710 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:00:11.031685 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6329017_16da_4de4_aee0_14082b9f0a0f.slice/crio-1fb58c555b240af5c38aa1cb32e14e16232735b5271be2f5fa4b48c528d92478 WatchSource:0}: Error finding container 1fb58c555b240af5c38aa1cb32e14e16232735b5271be2f5fa4b48c528d92478: Status 404 returned error can't find the container with id 1fb58c555b240af5c38aa1cb32e14e16232735b5271be2f5fa4b48c528d92478 Sep 11 18:00:11.033587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:11.033569 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:00:11.051382 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:11.051357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" event={"ID":"b6329017-16da-4de4-aee0-14082b9f0a0f","Type":"ContainerStarted","Data":"1fb58c555b240af5c38aa1cb32e14e16232735b5271be2f5fa4b48c528d92478"} Sep 11 18:00:12.042274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.042235 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.135903 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.135861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.238431 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.238387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.522672 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.522632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.670209 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.670171 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v"] Sep 11 18:00:12.673880 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.673860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.679139 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.679102 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 18:00:12.681447 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.681407 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srmfv\"" Sep 11 18:00:12.681570 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.681494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 18:00:12.682064 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.682044 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 18:00:12.682809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.682377 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 18:00:12.699190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.699162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.721397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.721365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v"] Sep 11 18:00:12.733903 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.733877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47fd9dbe-66b1-4263-9dd6-1297bc874315-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.734064 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.733910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47fd9dbe-66b1-4263-9dd6-1297bc874315-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.734064 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.733941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47fd9dbe-66b1-4263-9dd6-1297bc874315-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.734064 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.733963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6j9\" (UniqueName: \"kubernetes.io/projected/47fd9dbe-66b1-4263-9dd6-1297bc874315-kube-api-access-qk6j9\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.734064 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.734004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47fd9dbe-66b1-4263-9dd6-1297bc874315-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.834852 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.834758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47fd9dbe-66b1-4263-9dd6-1297bc874315-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.834852 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.834801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47fd9dbe-66b1-4263-9dd6-1297bc874315-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.834852 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.834835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47fd9dbe-66b1-4263-9dd6-1297bc874315-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.835119 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.834856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6j9\" (UniqueName: \"kubernetes.io/projected/47fd9dbe-66b1-4263-9dd6-1297bc874315-kube-api-access-qk6j9\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.835119 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.834922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47fd9dbe-66b1-4263-9dd6-1297bc874315-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.835408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.835374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47fd9dbe-66b1-4263-9dd6-1297bc874315-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.837555 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.837526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47fd9dbe-66b1-4263-9dd6-1297bc874315-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.837921 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.837899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47fd9dbe-66b1-4263-9dd6-1297bc874315-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.838015 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.837947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47fd9dbe-66b1-4263-9dd6-1297bc874315-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.844202 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.844173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6j9\" (UniqueName: \"kubernetes.io/projected/47fd9dbe-66b1-4263-9dd6-1297bc874315-kube-api-access-qk6j9\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v\" (UID: \"47fd9dbe-66b1-4263-9dd6-1297bc874315\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:12.850654 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.850629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.958541 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.958506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:12.987124 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:12.987090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:13.062306 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:13.062268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:13.168865 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:13.168833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:13.263205 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:13.263171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:13.362028 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:13.361996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:14.016194 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:00:14.016162 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fd9dbe_66b1_4263_9dd6_1297bc874315.slice/crio-f15fdbf3b1adeac2120f2dea9f952959f47f815ee7ad36f391789f1ebf1ff1b4 WatchSource:0}: Error finding container f15fdbf3b1adeac2120f2dea9f952959f47f815ee7ad36f391789f1ebf1ff1b4: Status 404 returned error can't find the container with id f15fdbf3b1adeac2120f2dea9f952959f47f815ee7ad36f391789f1ebf1ff1b4 Sep 11 18:00:14.017783 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:14.017760 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v"] Sep 11 18:00:14.067437 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:14.067380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" event={"ID":"47fd9dbe-66b1-4263-9dd6-1297bc874315","Type":"ContainerStarted","Data":"f15fdbf3b1adeac2120f2dea9f952959f47f815ee7ad36f391789f1ebf1ff1b4"} Sep 11 18:00:14.069115 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:14.069082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" event={"ID":"b6329017-16da-4de4-aee0-14082b9f0a0f","Type":"ContainerStarted","Data":"b2154489d3df4e0e7d26c2203f655879430679fde3122843457ffb10ea582972"} Sep 11 18:00:14.069337 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:14.069319 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:14.108715 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:14.108666 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" podStartSLOduration=1.341360557 podStartE2EDuration="4.108649471s" totalImagesPullingTime="2.767288914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:14.107983935 +0000 UTC m=+358.139045190" watchObservedRunningTime="2026-09-11 18:00:14.108649471 +0000 UTC m=+358.139710701" Sep 11 18:00:17.086758 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:17.086669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" event={"ID":"47fd9dbe-66b1-4263-9dd6-1297bc874315","Type":"ContainerStarted","Data":"4cc9020f927dc3bfc9d9b27e01b008b79868f098eb0444cb68a700eb934edc36"} Sep 11 18:00:18.092858 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:18.092817 2579 generic.go:356] "Generic (PLEG): container finished" podID="47fd9dbe-66b1-4263-9dd6-1297bc874315" containerID="4cc9020f927dc3bfc9d9b27e01b008b79868f098eb0444cb68a700eb934edc36" exitCode=0 Sep 11 18:00:18.093234 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:18.092871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" event={"ID":"47fd9dbe-66b1-4263-9dd6-1297bc874315","Type":"ContainerDied","Data":"4cc9020f927dc3bfc9d9b27e01b008b79868f098eb0444cb68a700eb934edc36"} Sep 11 18:00:19.099556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:19.099513 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" event={"ID":"47fd9dbe-66b1-4263-9dd6-1297bc874315","Type":"ContainerStarted","Data":"e726378020ed87bce877d12e8b738b85519812d15553dc6a4dd20354ecc5c3f4"} Sep 11 18:00:19.100026 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:19.099604 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:19.145128 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:19.145065 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" podStartSLOduration=2.884348843 podStartE2EDuration="7.14504052s" totalImagesPullingTime="3.167989888s" totalInitContainerRuntime="1.092701789s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:19.144049008 +0000 UTC m=+363.175110258" watchObservedRunningTime="2026-09-11 18:00:19.14504052 +0000 UTC m=+363.176101752" Sep 11 18:00:25.077548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:25.077515 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-jc9pj" Sep 11 18:00:30.106010 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:30.105979 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v" Sep 11 18:00:33.112769 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:33.112730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:33.283816 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:33.283771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:33.508944 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:33.508910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:33.638735 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:33.638703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:35.527900 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:35.527867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:35.628139 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:35.628106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:36.348234 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:36.348204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:43.320270 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.320235 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:43.420206 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.420177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:43.536710 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.536668 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87"] Sep 11 18:00:43.539012 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.538987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.542433 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.542392 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vxhbh\"" Sep 11 18:00:43.542566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.542473 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 18:00:43.554094 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.554071 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87"] Sep 11 18:00:43.574865 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.574802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:43.681626 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.681591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:43.706237 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706237 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7wj\" (UniqueName: \"kubernetes.io/projected/43ee41ff-9811-4d0c-b51d-968f357d8887-kube-api-access-5w7wj\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.706511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.706495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee41ff-9811-4d0c-b51d-968f357d8887-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.775868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.775831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:43.807313 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7wj\" (UniqueName: \"kubernetes.io/projected/43ee41ff-9811-4d0c-b51d-968f357d8887-kube-api-access-5w7wj\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee41ff-9811-4d0c-b51d-968f357d8887-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807794 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807794 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807794 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.807942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.807817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.808039 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.808016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.808245 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.808219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.808364 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.808257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.808573 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.808550 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ee41ff-9811-4d0c-b51d-968f357d8887-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.810153 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.810128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.810916 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.810893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.815958 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.815931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ee41ff-9811-4d0c-b51d-968f357d8887-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.816547 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.816513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7wj\" (UniqueName: \"kubernetes.io/projected/43ee41ff-9811-4d0c-b51d-968f357d8887-kube-api-access-5w7wj\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvgj87\" (UID: \"43ee41ff-9811-4d0c-b51d-968f357d8887\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:43.851568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:43.851517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:44.073865 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.073828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:44.131380 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.131268 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87"] Sep 11 18:00:44.133377 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:00:44.133335 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ee41ff_9811_4d0c_b51d_968f357d8887.slice/crio-3b25f276e16cd005e960aea4fa9f58eaa46f9555a0d87e2608dd43f1d8c6d608 WatchSource:0}: Error finding container 3b25f276e16cd005e960aea4fa9f58eaa46f9555a0d87e2608dd43f1d8c6d608: Status 404 returned error can't find the container with id 3b25f276e16cd005e960aea4fa9f58eaa46f9555a0d87e2608dd43f1d8c6d608 Sep 11 18:00:44.162509 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.162482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:44.216192 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.216153 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" event={"ID":"43ee41ff-9811-4d0c-b51d-968f357d8887","Type":"ContainerStarted","Data":"3b25f276e16cd005e960aea4fa9f58eaa46f9555a0d87e2608dd43f1d8c6d608"} Sep 11 18:00:44.264400 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.264359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:44.443599 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.443491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:44.865501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.865354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:44.943961 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:44.943929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:47.157860 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:47.157823 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:00:47.158166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:47.157897 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:00:47.158166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:47.157925 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:00:47.258006 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:47.257967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" event={"ID":"43ee41ff-9811-4d0c-b51d-968f357d8887","Type":"ContainerStarted","Data":"a83c0f286e3df0758a406f5db826fbb78348de48ae19c0704cc313a6ebdeb88d"} Sep 11 18:00:47.285194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:47.285133 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" podStartSLOduration=1.263430254 podStartE2EDuration="4.285107864s" totalImagesPullingTime="3.02167761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:47.282585329 +0000 UTC m=+391.313646586" watchObservedRunningTime="2026-09-11 18:00:47.285107864 +0000 UTC m=+391.316169094" Sep 11 18:00:47.852658 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:47.852622 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:48.856514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:48.856485 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:49.269676 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:49.269643 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:49.270827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:49.270803 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvgj87" Sep 11 18:00:51.822463 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.822429 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr"] Sep 11 18:00:51.825349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.825331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.829171 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.829154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 18:00:51.829269 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.829249 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 18:00:51.829328 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.829247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 18:00:51.829650 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.829632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zsf55\"" Sep 11 18:00:51.842201 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.842179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr"] Sep 11 18:00:51.977169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.977169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1990330b-607a-43a2-8990-fe64a6ae016e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.977378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1990330b-607a-43a2-8990-fe64a6ae016e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.977378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.977378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wjg\" (UniqueName: \"kubernetes.io/projected/1990330b-607a-43a2-8990-fe64a6ae016e-kube-api-access-76wjg\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.977378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1990330b-607a-43a2-8990-fe64a6ae016e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:51.977378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:51.977348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1990330b-607a-43a2-8990-fe64a6ae016e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1990330b-607a-43a2-8990-fe64a6ae016e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wjg\" (UniqueName: \"kubernetes.io/projected/1990330b-607a-43a2-8990-fe64a6ae016e-kube-api-access-76wjg\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1990330b-607a-43a2-8990-fe64a6ae016e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.078560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.078451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.079333 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.079305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1990330b-607a-43a2-8990-fe64a6ae016e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.080805 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.080781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1990330b-607a-43a2-8990-fe64a6ae016e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.081177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.081148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.081409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.081387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.081470 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.081442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1990330b-607a-43a2-8990-fe64a6ae016e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.091084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.091066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1990330b-607a-43a2-8990-fe64a6ae016e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.108529 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.108506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wjg\" (UniqueName: \"kubernetes.io/projected/1990330b-607a-43a2-8990-fe64a6ae016e-kube-api-access-76wjg\") pod \"istiod-openshift-gateway-5677ccb49b-r7bxr\" (UID: \"1990330b-607a-43a2-8990-fe64a6ae016e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.134139 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.134119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:52.413783 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:52.413751 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr"] Sep 11 18:00:52.418524 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:00:52.418498 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1990330b_607a_43a2_8990_fe64a6ae016e.slice/crio-29c9d87614dd0c654b28d987d44ca85571fe4606f40f25b145c5ca81e5cb828d WatchSource:0}: Error finding container 29c9d87614dd0c654b28d987d44ca85571fe4606f40f25b145c5ca81e5cb828d: Status 404 returned error can't find the container with id 29c9d87614dd0c654b28d987d44ca85571fe4606f40f25b145c5ca81e5cb828d Sep 11 18:00:53.293243 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:53.293209 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" event={"ID":"1990330b-607a-43a2-8990-fe64a6ae016e","Type":"ContainerStarted","Data":"29c9d87614dd0c654b28d987d44ca85571fe4606f40f25b145c5ca81e5cb828d"} Sep 11 18:00:54.811366 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:54.811283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:55.028590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:55.028553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:55.250209 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:55.250170 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:00:55.250309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:55.250242 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:00:55.295391 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:55.295364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:55.650668 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:55.650626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:56.265727 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.265693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:56.313896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.313862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" event={"ID":"1990330b-607a-43a2-8990-fe64a6ae016e","Type":"ContainerStarted","Data":"06bd90dbaf579cb0a55d68b998f3c7fd3bd20c7287d997ce6830b8f0547513a6"} Sep 11 18:00:56.314106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.314087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:56.315874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.315852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" Sep 11 18:00:56.383587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.383554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:00:56.408227 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.408177 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r7bxr" podStartSLOduration=2.579044053 podStartE2EDuration="5.40816072s" totalImagesPullingTime="2.829116667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:56.361798695 +0000 UTC m=+400.392859925" watchObservedRunningTime="2026-09-11 18:00:56.40816072 +0000 UTC m=+400.439221950" Sep 11 18:00:56.665506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:00:56.665463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:08.007224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:08.007185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:08.117771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:08.117738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:37.609242 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:37.609150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:37.828185 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:37.828152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:39.396480 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:39.396440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:39.657744 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:39.657661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:45.250834 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:45.250801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:01:45.353033 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:01:45.353000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:00.027040 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:00.027004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:00.826670 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:00.826634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:01.630540 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:01.630511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:03.870234 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:03.870199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:03.973776 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:03.973718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:05.237779 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:05.237741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:06.281143 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:06.281106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:07.026590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:07.026559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:07.446433 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:07.446395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:09.334216 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:09.334185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:09.440952 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:09.440915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:10.375645 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:10.375606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:11.095374 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:11.095344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:13.014655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:13.014622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:13.930357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:13.930327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:14.504377 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:14.504349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:15.035509 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:15.035481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:15.136460 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:15.136410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:15.428115 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:15.428081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:15.527991 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:15.527957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:15.835859 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:15.835785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:15.938392 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:15.938359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:16.382055 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.382022 2579 scope.go:117] "RemoveContainer" containerID="fed5a889bf0fda6634c2da63034d77b1b7974621c961835e0e4f09d6b2356b0d" Sep 11 18:02:16.391325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.391306 2579 scope.go:117] "RemoveContainer" containerID="20a6c76da429df16bc2c12edfb0273abbeed64a5aa97e74ac5e972c4d55692de" Sep 11 18:02:16.398934 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.398907 2579 scope.go:117] "RemoveContainer" containerID="4f0700ca72f80d62df103401b63c52fce754f3e5765f734b76431728f8fdf33a" Sep 11 18:02:16.405270 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.405248 2579 scope.go:117] "RemoveContainer" containerID="a1ca09739bd72c557854583b886008fb33c82fd22b818ca12c1315fad6b333b7" Sep 11 18:02:16.411997 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.411983 2579 scope.go:117] "RemoveContainer" containerID="3f360a07d9f55e9f26d92285b9c837be02a9732723cb2a29dcb1f2a3342ea1ec" Sep 11 18:02:16.422277 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.422257 2579 scope.go:117] "RemoveContainer" containerID="22c6266f830d891ee5033f602b37815f2370ab712eb71012482266516f3276dc" Sep 11 18:02:16.430548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:16.430527 2579 scope.go:117] "RemoveContainer" containerID="85ff76737a888a5b7b8cb9b8a236d251f9a6311417ac086b9e489a3966b1ed71" Sep 11 18:02:21.532057 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:21.532023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:22.771737 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:22.771709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:28.113487 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:28.113449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:28.279610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:28.279571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:31.103578 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:31.103543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:31.210512 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:31.210476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:34.694867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:34.694835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:35.064140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:35.064050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:36.335206 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:36.335177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:36.475867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:36.475842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:36.669913 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:36.669879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:36.799713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:36.799680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:42.551857 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:42.551822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:42.669479 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:42.669445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:46.809240 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:46.809212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:46.927729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:46.927699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:49.050089 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:49.050046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:49.206990 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:49.206957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:52.033375 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:52.033335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:52.161119 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:52.161082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:52.387712 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:52.387678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:52.487707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:52.487676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:58.430198 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:58.430165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:02:58.526914 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:02:58.526885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:18.261103 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:18.261019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:18.375022 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:18.374988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:18.800689 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:18.800658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:25.002995 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:25.002961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:31.554481 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:31.554448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:31.896184 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:31.896143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:38.942610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:38.942572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:39.029249 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.029214 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67cfdfffd9-bgdvp"] Sep 11 18:03:39.032523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.032502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.035697 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.035675 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 18:03:39.035697 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.035685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 18:03:39.035856 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.035727 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 18:03:39.037327 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.037310 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vtvpv\"" Sep 11 18:03:39.042690 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.042672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67cfdfffd9-bgdvp"] Sep 11 18:03:39.073343 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.073312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:39.114364 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.114334 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9b9f55b89-nvx9v"] Sep 11 18:03:39.117472 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.117456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.126956 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.126920 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-nvx9v"] Sep 11 18:03:39.146329 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.146297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-metrics-certs\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.146490 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.146371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-webhook-cert\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.146490 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.146401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc5c\" (UniqueName: \"kubernetes.io/projected/9278ebe3-689c-4ce7-9e20-0c34793ada3e-kube-api-access-fnc5c\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.247467 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.247370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-metrics-certs\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.247621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.247478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9klrf\" (UniqueName: \"kubernetes.io/projected/923d603b-83c2-402f-acb6-4a85c45117a3-kube-api-access-9klrf\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.247621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.247553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-webhook-cert\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.247621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.247578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-webhook-cert\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.247781 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.247644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc5c\" (UniqueName: \"kubernetes.io/projected/9278ebe3-689c-4ce7-9e20-0c34793ada3e-kube-api-access-fnc5c\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.247781 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.247733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-metrics-certs\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.249972 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.249946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-metrics-certs\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.250095 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.250076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-webhook-cert\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.258933 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.258906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc5c\" (UniqueName: \"kubernetes.io/projected/9278ebe3-689c-4ce7-9e20-0c34793ada3e-kube-api-access-fnc5c\") pod \"maas-controller-67cfdfffd9-bgdvp\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.264092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.264070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:39.343670 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.343637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:39.348865 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.348845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-webhook-cert\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.348941 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.348896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-metrics-certs\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.348941 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.348930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9klrf\" (UniqueName: \"kubernetes.io/projected/923d603b-83c2-402f-acb6-4a85c45117a3-kube-api-access-9klrf\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.351376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.351351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-webhook-cert\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.351579 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.351559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-metrics-certs\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.357621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.357592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9klrf\" (UniqueName: \"kubernetes.io/projected/923d603b-83c2-402f-acb6-4a85c45117a3-kube-api-access-9klrf\") pod \"maas-controller-9b9f55b89-nvx9v\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.428745 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.428711 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:39.447240 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.447214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:39.617805 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.617773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67cfdfffd9-bgdvp"] Sep 11 18:03:39.621563 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:03:39.621538 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9278ebe3_689c_4ce7_9e20_0c34793ada3e.slice/crio-ff4ff1d9a5cfdb8a20187ecd8e5118911e1f114cbd7d6dc9045e0f17cf498035 WatchSource:0}: Error finding container ff4ff1d9a5cfdb8a20187ecd8e5118911e1f114cbd7d6dc9045e0f17cf498035: Status 404 returned error can't find the container with id ff4ff1d9a5cfdb8a20187ecd8e5118911e1f114cbd7d6dc9045e0f17cf498035 Sep 11 18:03:39.704084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:39.704062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-nvx9v"] Sep 11 18:03:39.706017 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:03:39.705988 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923d603b_83c2_402f_acb6_4a85c45117a3.slice/crio-46c31eaab3e49e558e9be2dd79c2b389855428ff9aa7471a9cd30188a3135b06 WatchSource:0}: Error finding container 46c31eaab3e49e558e9be2dd79c2b389855428ff9aa7471a9cd30188a3135b06: Status 404 returned error can't find the container with id 46c31eaab3e49e558e9be2dd79c2b389855428ff9aa7471a9cd30188a3135b06 Sep 11 18:03:40.025993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:40.025902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" event={"ID":"9278ebe3-689c-4ce7-9e20-0c34793ada3e","Type":"ContainerStarted","Data":"ff4ff1d9a5cfdb8a20187ecd8e5118911e1f114cbd7d6dc9045e0f17cf498035"} Sep 11 18:03:40.027070 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:40.027045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" event={"ID":"923d603b-83c2-402f-acb6-4a85c45117a3","Type":"ContainerStarted","Data":"46c31eaab3e49e558e9be2dd79c2b389855428ff9aa7471a9cd30188a3135b06"} Sep 11 18:03:42.511048 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:42.511011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:42.635556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:42.635524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:03:43.049967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:43.049859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" event={"ID":"923d603b-83c2-402f-acb6-4a85c45117a3","Type":"ContainerStarted","Data":"9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0"} Sep 11 18:03:43.049967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:43.049937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:43.051334 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:43.051308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" event={"ID":"9278ebe3-689c-4ce7-9e20-0c34793ada3e","Type":"ContainerStarted","Data":"314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1"} Sep 11 18:03:43.051483 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:43.051360 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:43.098995 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:43.098937 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" podStartSLOduration=1.530384569 podStartE2EDuration="4.098919796s" totalImagesPullingTime="2.568535227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:43.071152971 +0000 UTC m=+567.102214226" watchObservedRunningTime="2026-09-11 18:03:43.098919796 +0000 UTC m=+567.129981038" Sep 11 18:03:43.100092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:43.100062 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" podStartSLOduration=1.437232566 podStartE2EDuration="4.100051213s" totalImagesPullingTime="2.662818647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:43.098121621 +0000 UTC m=+567.129182843" watchObservedRunningTime="2026-09-11 18:03:43.100051213 +0000 UTC m=+567.131112502" Sep 11 18:03:54.059292 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.059262 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:54.059775 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.059332 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:03:54.132558 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.132524 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67cfdfffd9-bgdvp"] Sep 11 18:03:54.132779 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.132738 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" podUID="9278ebe3-689c-4ce7-9e20-0c34793ada3e" containerName="manager" containerID="cri-o://314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1" gracePeriod=30 Sep 11 18:03:54.376100 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.376078 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:54.487086 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.487055 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-metrics-certs\") pod \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " Sep 11 18:03:54.487272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.487141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnc5c\" (UniqueName: \"kubernetes.io/projected/9278ebe3-689c-4ce7-9e20-0c34793ada3e-kube-api-access-fnc5c\") pod \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " Sep 11 18:03:54.487272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.487183 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-webhook-cert\") pod \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\" (UID: \"9278ebe3-689c-4ce7-9e20-0c34793ada3e\") " Sep 11 18:03:54.489388 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.489351 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9278ebe3-689c-4ce7-9e20-0c34793ada3e" (UID: "9278ebe3-689c-4ce7-9e20-0c34793ada3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:03:54.489536 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.489437 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9278ebe3-689c-4ce7-9e20-0c34793ada3e" (UID: "9278ebe3-689c-4ce7-9e20-0c34793ada3e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:03:54.489536 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.489480 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9278ebe3-689c-4ce7-9e20-0c34793ada3e-kube-api-access-fnc5c" (OuterVolumeSpecName: "kube-api-access-fnc5c") pod "9278ebe3-689c-4ce7-9e20-0c34793ada3e" (UID: "9278ebe3-689c-4ce7-9e20-0c34793ada3e"). InnerVolumeSpecName "kube-api-access-fnc5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:03:54.588898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.588747 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnc5c\" (UniqueName: \"kubernetes.io/projected/9278ebe3-689c-4ce7-9e20-0c34793ada3e-kube-api-access-fnc5c\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:03:54.588898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.588779 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-webhook-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:03:54.588898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:54.588795 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9278ebe3-689c-4ce7-9e20-0c34793ada3e-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:03:55.106559 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.106528 2579 generic.go:356] "Generic (PLEG): container finished" podID="9278ebe3-689c-4ce7-9e20-0c34793ada3e" containerID="314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1" exitCode=0 Sep 11 18:03:55.107019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.106577 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" Sep 11 18:03:55.107019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.106614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" event={"ID":"9278ebe3-689c-4ce7-9e20-0c34793ada3e","Type":"ContainerDied","Data":"314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1"} Sep 11 18:03:55.107019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.106687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67cfdfffd9-bgdvp" event={"ID":"9278ebe3-689c-4ce7-9e20-0c34793ada3e","Type":"ContainerDied","Data":"ff4ff1d9a5cfdb8a20187ecd8e5118911e1f114cbd7d6dc9045e0f17cf498035"} Sep 11 18:03:55.107019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.106732 2579 scope.go:117] "RemoveContainer" containerID="314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1" Sep 11 18:03:55.115984 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.115957 2579 scope.go:117] "RemoveContainer" containerID="314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1" Sep 11 18:03:55.116272 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:03:55.116252 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1\": container with ID starting with 314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1 not found: ID does not exist" containerID="314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1" Sep 11 18:03:55.116345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.116295 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1"} err="failed to get container status \"314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1\": rpc error: code = NotFound desc = could not find container \"314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1\": container with ID starting with 314591c3678799003bdbec62c2ef82238160650e1b24c652744752c53d98d1b1 not found: ID does not exist" Sep 11 18:03:55.131806 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.131778 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67cfdfffd9-bgdvp"] Sep 11 18:03:55.137680 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:55.137655 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67cfdfffd9-bgdvp"] Sep 11 18:03:56.444034 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:03:56.443991 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9278ebe3-689c-4ce7-9e20-0c34793ada3e" path="/var/lib/kubelet/pods/9278ebe3-689c-4ce7-9e20-0c34793ada3e/volumes" Sep 11 18:04:04.286371 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:04.286329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:04.388101 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:04.388064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:12.899753 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:12.899712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:13.043159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:13.043117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:14.394211 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.394177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:14.437928 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.437860 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw"] Sep 11 18:04:14.438285 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.438269 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9278ebe3-689c-4ce7-9e20-0c34793ada3e" containerName="manager" Sep 11 18:04:14.438365 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.438287 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9278ebe3-689c-4ce7-9e20-0c34793ada3e" containerName="manager" Sep 11 18:04:14.438365 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.438344 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9278ebe3-689c-4ce7-9e20-0c34793ada3e" containerName="manager" Sep 11 18:04:14.440488 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.440468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.444017 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.443984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 18:04:14.444140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.444047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 18:04:14.444233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.444182 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 18:04:14.444412 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.444394 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 18:04:14.452513 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.452492 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw"] Sep 11 18:04:14.547046 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.547010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-maas-api-tls\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.547233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.547068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224e23ce-68eb-4eb9-8cff-e799380085df-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.547233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.547133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkgh\" (UniqueName: \"kubernetes.io/projected/224e23ce-68eb-4eb9-8cff-e799380085df-kube-api-access-kgkgh\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.547233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.547204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-metrics-certs\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.628079 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.628044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:14.648389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.648317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224e23ce-68eb-4eb9-8cff-e799380085df-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.648389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.648355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkgh\" (UniqueName: \"kubernetes.io/projected/224e23ce-68eb-4eb9-8cff-e799380085df-kube-api-access-kgkgh\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.648602 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.648453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-metrics-certs\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.648602 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.648510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-maas-api-tls\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.649034 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.649015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224e23ce-68eb-4eb9-8cff-e799380085df-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.651033 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.651015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-maas-api-tls\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.651188 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.651165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-metrics-certs\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.657809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.657783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkgh\" (UniqueName: \"kubernetes.io/projected/224e23ce-68eb-4eb9-8cff-e799380085df-kube-api-access-kgkgh\") pod \"maas-api-66ff6b6fc9-q8qmw\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.752525 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.752498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:14.772379 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:14.772347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:15.033367 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.033342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw"] Sep 11 18:04:15.035652 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:04:15.035626 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224e23ce_68eb_4eb9_8cff_e799380085df.slice/crio-522177bc3041ca2b068763f84d2a0734915b2b99ebf2d71643fe04f892620b19 WatchSource:0}: Error finding container 522177bc3041ca2b068763f84d2a0734915b2b99ebf2d71643fe04f892620b19: Status 404 returned error can't find the container with id 522177bc3041ca2b068763f84d2a0734915b2b99ebf2d71643fe04f892620b19 Sep 11 18:04:15.082608 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.082573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:15.171657 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.171628 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk"] Sep 11 18:04:15.174788 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.174773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.177855 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.177835 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4r4rh\"" Sep 11 18:04:15.177974 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.177857 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 18:04:15.185194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.185172 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk"] Sep 11 18:04:15.197005 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.196972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" event={"ID":"224e23ce-68eb-4eb9-8cff-e799380085df","Type":"ContainerStarted","Data":"522177bc3041ca2b068763f84d2a0734915b2b99ebf2d71643fe04f892620b19"} Sep 11 18:04:15.254729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.254696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a694aa-f5b7-40dc-9c90-3869ab675962-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dnhtk\" (UID: \"45a694aa-f5b7-40dc-9c90-3869ab675962\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.254883 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.254821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9w6\" (UniqueName: \"kubernetes.io/projected/45a694aa-f5b7-40dc-9c90-3869ab675962-kube-api-access-7q9w6\") pod \"payload-pre-processing-59fbc47bf8-dnhtk\" (UID: \"45a694aa-f5b7-40dc-9c90-3869ab675962\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.356011 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.355914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a694aa-f5b7-40dc-9c90-3869ab675962-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dnhtk\" (UID: \"45a694aa-f5b7-40dc-9c90-3869ab675962\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.356011 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.355994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9w6\" (UniqueName: \"kubernetes.io/projected/45a694aa-f5b7-40dc-9c90-3869ab675962-kube-api-access-7q9w6\") pod \"payload-pre-processing-59fbc47bf8-dnhtk\" (UID: \"45a694aa-f5b7-40dc-9c90-3869ab675962\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.356976 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.356951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45a694aa-f5b7-40dc-9c90-3869ab675962-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dnhtk\" (UID: \"45a694aa-f5b7-40dc-9c90-3869ab675962\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.365720 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.365696 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9w6\" (UniqueName: \"kubernetes.io/projected/45a694aa-f5b7-40dc-9c90-3869ab675962-kube-api-access-7q9w6\") pod \"payload-pre-processing-59fbc47bf8-dnhtk\" (UID: \"45a694aa-f5b7-40dc-9c90-3869ab675962\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.487238 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.487204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:15.778138 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:04:15.778097 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a694aa_f5b7_40dc_9c90_3869ab675962.slice/crio-dacbca4345732989e3d602bc36dd99a942d564463e33e72bd47609371aac1826 WatchSource:0}: Error finding container dacbca4345732989e3d602bc36dd99a942d564463e33e72bd47609371aac1826: Status 404 returned error can't find the container with id dacbca4345732989e3d602bc36dd99a942d564463e33e72bd47609371aac1826 Sep 11 18:04:15.778475 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:15.778452 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk"] Sep 11 18:04:16.202577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:16.202538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" event={"ID":"45a694aa-f5b7-40dc-9c90-3869ab675962","Type":"ContainerStarted","Data":"dacbca4345732989e3d602bc36dd99a942d564463e33e72bd47609371aac1826"} Sep 11 18:04:16.555958 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:16.555923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:17.209536 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:17.209438 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" event={"ID":"224e23ce-68eb-4eb9-8cff-e799380085df","Type":"ContainerStarted","Data":"407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4"} Sep 11 18:04:17.209710 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:17.209651 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:17.232040 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:17.231984 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" podStartSLOduration=1.285968747 podStartE2EDuration="3.23196649s" totalImagesPullingTime="1.945997743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:17.230729318 +0000 UTC m=+601.261790576" watchObservedRunningTime="2026-09-11 18:04:17.23196649 +0000 UTC m=+601.263027719" Sep 11 18:04:19.221410 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:19.221368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" event={"ID":"45a694aa-f5b7-40dc-9c90-3869ab675962","Type":"ContainerStarted","Data":"fa5619dfa572bd654b4895dc4f3de8140f8b6188ed5e7e3cb435b93189d4b5a1"} Sep 11 18:04:19.221832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:19.221471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:19.242871 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:19.242819 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" podStartSLOduration=1.75336817 podStartE2EDuration="4.242801993s" totalImagesPullingTime="2.489433823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:19.240998326 +0000 UTC m=+603.272059556" watchObservedRunningTime="2026-09-11 18:04:19.242801993 +0000 UTC m=+603.273863246" Sep 11 18:04:23.220584 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:23.220558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:04:24.402241 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:24.402215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:24.504362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:24.504319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:30.227759 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:30.227730 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dnhtk" Sep 11 18:04:33.628728 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:33.628694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:48.572907 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.572829 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw"] Sep 11 18:04:48.577309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.577288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.582965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.582942 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 18:04:48.582965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.582956 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 18:04:48.583148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.582954 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d5n2t\"" Sep 11 18:04:48.583148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.582951 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 11 18:04:48.596999 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.596975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw"] Sep 11 18:04:48.656618 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.656582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.656778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.656641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.656778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.656698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.656778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.656729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxqb\" (UniqueName: \"kubernetes.io/projected/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-kube-api-access-lfxqb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.656778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.656755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.656924 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.656788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.686604 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.686566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:04:48.757568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.757724 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.757724 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.757724 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxqb\" (UniqueName: \"kubernetes.io/projected/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-kube-api-access-lfxqb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.757724 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.757724 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.758020 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.757999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.758095 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.758024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.758151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.758115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.760060 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.760041 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.760183 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.760166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.768643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.768620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxqb\" (UniqueName: \"kubernetes.io/projected/74cd62a3-ddff-4fe7-bc66-d1165e22e3cc-kube-api-access-lfxqb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nvwkw\" (UID: \"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:48.887570 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:48.887530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:49.166656 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:49.166569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw"] Sep 11 18:04:49.169274 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:04:49.169232 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cd62a3_ddff_4fe7_bc66_d1165e22e3cc.slice/crio-21cf53e33ae93a8b97e93e3c722c428e6643c87ab8ca210aaf430c728967d95d WatchSource:0}: Error finding container 21cf53e33ae93a8b97e93e3c722c428e6643c87ab8ca210aaf430c728967d95d: Status 404 returned error can't find the container with id 21cf53e33ae93a8b97e93e3c722c428e6643c87ab8ca210aaf430c728967d95d Sep 11 18:04:49.352225 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:49.352186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" event={"ID":"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc","Type":"ContainerStarted","Data":"21cf53e33ae93a8b97e93e3c722c428e6643c87ab8ca210aaf430c728967d95d"} Sep 11 18:04:50.358543 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:50.358508 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" event={"ID":"74cd62a3-ddff-4fe7-bc66-d1165e22e3cc","Type":"ContainerStarted","Data":"71cb7d02d5ed4434d7a2b7a28cc27d2214efcde0b76f2b0c8e62c1bba3beab60"} Sep 11 18:04:50.381814 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:50.381756 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" podStartSLOduration=1.416588534 podStartE2EDuration="2.381736737s" totalImagesPullingTime="965.148203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:50.379730615 +0000 UTC m=+634.410791873" watchObservedRunningTime="2026-09-11 18:04:50.381736737 +0000 UTC m=+634.412797968" Sep 11 18:04:58.887989 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:58.887932 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:58.887989 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:58.888010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:58.901113 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:58.901085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:04:59.407832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:04:59.407797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nvwkw" Sep 11 18:05:02.319387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:02.319356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:05:16.774342 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:16.774303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:05:24.619946 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.619913 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8"] Sep 11 18:05:24.624440 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.624407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.627866 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.627848 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 18:05:24.642468 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.642445 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8"] Sep 11 18:05:24.678551 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.678524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:05:24.790802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.790769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.790970 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.790817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.790970 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.790844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxs2\" (UniqueName: \"kubernetes.io/projected/b2ca90fc-31f5-4375-8c1b-155912f7ea56-kube-api-access-bdxs2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.790970 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.790951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ca90fc-31f5-4375-8c1b-155912f7ea56-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.791126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.791016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.791126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.791046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892470 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892470 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxs2\" (UniqueName: \"kubernetes.io/projected/b2ca90fc-31f5-4375-8c1b-155912f7ea56-kube-api-access-bdxs2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892470 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ca90fc-31f5-4375-8c1b-155912f7ea56-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892470 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892668 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892726 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.892808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.892789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.894549 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.894530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ca90fc-31f5-4375-8c1b-155912f7ea56-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.894882 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.894866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ca90fc-31f5-4375-8c1b-155912f7ea56-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.903161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.903138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxs2\" (UniqueName: \"kubernetes.io/projected/b2ca90fc-31f5-4375-8c1b-155912f7ea56-kube-api-access-bdxs2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8\" (UID: \"b2ca90fc-31f5-4375-8c1b-155912f7ea56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:24.934310 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:24.934277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:25.210939 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:25.210912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8"] Sep 11 18:05:25.212783 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:05:25.212755 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ca90fc_31f5_4375_8c1b_155912f7ea56.slice/crio-fe85712a4aad294be445ac1c8dff479a1164ec4e202021a9b2267f9b68d7b919 WatchSource:0}: Error finding container fe85712a4aad294be445ac1c8dff479a1164ec4e202021a9b2267f9b68d7b919: Status 404 returned error can't find the container with id fe85712a4aad294be445ac1c8dff479a1164ec4e202021a9b2267f9b68d7b919 Sep 11 18:05:25.214166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:25.214139 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:05:25.516646 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:25.516617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" event={"ID":"b2ca90fc-31f5-4375-8c1b-155912f7ea56","Type":"ContainerStarted","Data":"a2d45e2e72d28e3d9911200fa449c9c57731800f625ab885dd273b413e1561dc"} Sep 11 18:05:25.516821 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:25.516667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" event={"ID":"b2ca90fc-31f5-4375-8c1b-155912f7ea56","Type":"ContainerStarted","Data":"fe85712a4aad294be445ac1c8dff479a1164ec4e202021a9b2267f9b68d7b919"} Sep 11 18:05:25.539709 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:25.539655 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" podStartSLOduration=1.34727819 podStartE2EDuration="1.539640308s" totalImagesPullingTime="192.362118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:05:25.538003909 +0000 UTC m=+669.569065161" watchObservedRunningTime="2026-09-11 18:05:25.539640308 +0000 UTC m=+669.570701537" Sep 11 18:05:34.934742 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:34.934701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:34.935274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:34.934784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:34.947946 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:34.947914 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:35.575400 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:35.575376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8" Sep 11 18:05:37.999481 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:37.999449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:05:40.911501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:40.911438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:05:41.029655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:41.029613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:05:58.699767 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:05:58.699723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:06:00.848591 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:06:00.848553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:06:12.946942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:06:12.946865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:06:13.048133 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:06:13.048080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:06:26.598878 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:06:26.598815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:06:59.076352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:06:59.076318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:06:59.220056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:06:59.220000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:00.264635 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:00.264598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:00.410038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:00.409997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:15.675431 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:15.675387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:15.825781 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:15.825752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:16.187883 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.187826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:16.429167 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.429135 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m"] Sep 11 18:07:16.437174 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.432487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.440518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.440408 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f8a2d4-openshift-default-dockercfg-ww9ff\"" Sep 11 18:07:16.445134 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.445110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:16.451769 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.451750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m"] Sep 11 18:07:16.505035 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-certs\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505207 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtt2x\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-kube-api-access-dtt2x\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505207 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-socket\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505207 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-envoy\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505207 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-credential-socket\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505450 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65873c38-8f3d-4261-972e-23c9b1d91664-istio-podinfo\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505450 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-data\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-istio-token\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.505566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.505544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65873c38-8f3d-4261-972e-23c9b1d91664-istiod-ca-cert\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.568269 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.568238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:16.606089 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65873c38-8f3d-4261-972e-23c9b1d91664-istiod-ca-cert\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.606262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-certs\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.606262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt2x\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-kube-api-access-dtt2x\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.606262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-socket\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.606605 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-certs\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.606926 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-socket\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.606926 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.606797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65873c38-8f3d-4261-972e-23c9b1d91664-istiod-ca-cert\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.607099 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.607076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-envoy\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.607224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.607184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-credential-socket\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.607606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.607477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-credential-socket\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.607704 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.607658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65873c38-8f3d-4261-972e-23c9b1d91664-istio-podinfo\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.607840 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.607824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-data\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.608147 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.608108 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-data\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.608333 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.608313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-istio-token\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.610334 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.610312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-envoy\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.610590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.610571 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65873c38-8f3d-4261-972e-23c9b1d91664-istio-podinfo\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.623332 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.623300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt2x\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-kube-api-access-dtt2x\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.624192 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.624172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-istio-token\") pod \"e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.711472 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.711392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:16.750872 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.750841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:16.827307 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:16.827280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:17.051006 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.050924 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m"] Sep 11 18:07:17.054646 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:17.054618 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65873c38_8f3d_4261_972e_23c9b1d91664.slice/crio-91a225af9a1be08660714257d6f969403738f1135f71f8b8636690039ae9cf48 WatchSource:0}: Error finding container 91a225af9a1be08660714257d6f969403738f1135f71f8b8636690039ae9cf48: Status 404 returned error can't find the container with id 91a225af9a1be08660714257d6f969403738f1135f71f8b8636690039ae9cf48 Sep 11 18:07:17.056774 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.056739 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:07:17.056864 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.056818 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:07:17.056864 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.056852 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:07:17.195687 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.195653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:17.452755 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.452724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:17.779951 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:17.779870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:18.020318 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:18.020277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" event={"ID":"65873c38-8f3d-4261-972e-23c9b1d91664","Type":"ContainerStarted","Data":"1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725"} Sep 11 18:07:18.020522 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:18.020373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" event={"ID":"65873c38-8f3d-4261-972e-23c9b1d91664","Type":"ContainerStarted","Data":"91a225af9a1be08660714257d6f969403738f1135f71f8b8636690039ae9cf48"} Sep 11 18:07:18.051077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:18.050977 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podStartSLOduration=2.050956546 podStartE2EDuration="2.050956546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:18.048712339 +0000 UTC m=+782.079773589" watchObservedRunningTime="2026-09-11 18:07:18.050956546 +0000 UTC m=+782.082017775" Sep 11 18:07:18.751796 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:18.751753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:18.757014 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:18.756990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:19.025534 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:19.025435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:19.026472 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:19.026451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:19.686628 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:19.686593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:19.931706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:19.931671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:20.117988 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:20.117956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:20.334405 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:20.334373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:20.551370 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:20.551284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:21.640557 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.640528 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp"] Sep 11 18:07:21.644264 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.644247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.650992 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.650887 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f8a2d4-serving-cert\"" Sep 11 18:07:21.651217 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.651173 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2sf5b\"" Sep 11 18:07:21.664908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.664884 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp"] Sep 11 18:07:21.750849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.750813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:21.764514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.764468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.764692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.764526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4614eb16-3474-45ce-a151-51bc8c239020-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.764692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.764575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-metrics-certs\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.764692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.764600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccph\" (UniqueName: \"kubernetes.io/projected/4614eb16-3474-45ce-a151-51bc8c239020-kube-api-access-7ccph\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.865553 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.865511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.865553 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.865556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4614eb16-3474-45ce-a151-51bc8c239020-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.865793 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.865589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-metrics-certs\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.865793 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.865614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccph\" (UniqueName: \"kubernetes.io/projected/4614eb16-3474-45ce-a151-51bc8c239020-kube-api-access-7ccph\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.866278 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.866253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4614eb16-3474-45ce-a151-51bc8c239020-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.868507 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.868484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.868637 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.868570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-metrics-certs\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.877885 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.877860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccph\" (UniqueName: \"kubernetes.io/projected/4614eb16-3474-45ce-a151-51bc8c239020-kube-api-access-7ccph\") pod \"maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:21.956337 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:21.956248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:22.018171 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.018131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:22.236887 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.236860 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp"] Sep 11 18:07:22.238559 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:22.238532 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4614eb16_3474_45ce_a151_51bc8c239020.slice/crio-96569170f39ee994451ba575c16350c6b1cdf1bc74c75ca210cdafeecb214f20 WatchSource:0}: Error finding container 96569170f39ee994451ba575c16350c6b1cdf1bc74c75ca210cdafeecb214f20: Status 404 returned error can't find the container with id 96569170f39ee994451ba575c16350c6b1cdf1bc74c75ca210cdafeecb214f20 Sep 11 18:07:22.393503 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.393472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:22.649041 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.649000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:22.845306 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.845277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:22.968643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.968572 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6"] Sep 11 18:07:22.972233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.972212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:22.975193 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.975174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4e0b0b-serving-cert\"" Sep 11 18:07:22.989771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.989733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6"] Sep 11 18:07:22.998572 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:22.998538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:23.047370 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.047334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" event={"ID":"4614eb16-3474-45ce-a151-51bc8c239020","Type":"ContainerStarted","Data":"fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661"} Sep 11 18:07:23.047590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.047384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" event={"ID":"4614eb16-3474-45ce-a151-51bc8c239020","Type":"ContainerStarted","Data":"96569170f39ee994451ba575c16350c6b1cdf1bc74c75ca210cdafeecb214f20"} Sep 11 18:07:23.047590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.047441 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:23.075087 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.074345 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" podStartSLOduration=1.588548335 podStartE2EDuration="2.074324415s" totalImagesPullingTime="485.77608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:23.073969667 +0000 UTC m=+787.105030909" watchObservedRunningTime="2026-09-11 18:07:23.074324415 +0000 UTC m=+787.105385643" Sep 11 18:07:23.076524 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.076502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-metrics-certs\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.076651 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.076582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcncx\" (UniqueName: \"kubernetes.io/projected/65947d2f-2289-4d28-a0aa-bd95f04ad62b-kube-api-access-qcncx\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.076651 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.076612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65947d2f-2289-4d28-a0aa-bd95f04ad62b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.076651 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.076646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.150887 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.150840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:23.177982 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.177946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcncx\" (UniqueName: \"kubernetes.io/projected/65947d2f-2289-4d28-a0aa-bd95f04ad62b-kube-api-access-qcncx\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.178116 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.177998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65947d2f-2289-4d28-a0aa-bd95f04ad62b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.178116 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.178047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.178238 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.178184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-metrics-certs\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.179158 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.179025 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65947d2f-2289-4d28-a0aa-bd95f04ad62b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.181410 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.181384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-metrics-certs\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.181750 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.181723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.193022 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.192998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcncx\" (UniqueName: \"kubernetes.io/projected/65947d2f-2289-4d28-a0aa-bd95f04ad62b-kube-api-access-qcncx\") pod \"maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.284894 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.284804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:23.285647 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.285625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:23.382732 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.382700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:23.570263 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.570240 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6"] Sep 11 18:07:23.571746 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:23.571724 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65947d2f_2289_4d28_a0aa_bd95f04ad62b.slice/crio-be516cdd2be4e65771cc552442cee03b6278f0b2150b9f33410135caf0deb604 WatchSource:0}: Error finding container be516cdd2be4e65771cc552442cee03b6278f0b2150b9f33410135caf0deb604: Status 404 returned error can't find the container with id be516cdd2be4e65771cc552442cee03b6278f0b2150b9f33410135caf0deb604 Sep 11 18:07:23.770114 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.770078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:23.898559 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.898529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:23.941525 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.941488 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs"] Sep 11 18:07:23.944545 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.944526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:23.948374 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.948357 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4e0b0b\"" Sep 11 18:07:23.948499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.948388 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4e0b0b-dockercfg-d86st\"" Sep 11 18:07:23.958578 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:23.958554 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs"] Sep 11 18:07:24.016587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.016315 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g"] Sep 11 18:07:24.021480 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.021453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.042079 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.042047 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g"] Sep 11 18:07:24.053561 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.053529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" event={"ID":"65947d2f-2289-4d28-a0aa-bd95f04ad62b","Type":"ContainerStarted","Data":"4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab"} Sep 11 18:07:24.053706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.053585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" event={"ID":"65947d2f-2289-4d28-a0aa-bd95f04ad62b","Type":"ContainerStarted","Data":"be516cdd2be4e65771cc552442cee03b6278f0b2150b9f33410135caf0deb604"} Sep 11 18:07:24.086448 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.086389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.086448 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.086448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8nwc\" (UniqueName: \"kubernetes.io/projected/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-kube-api-access-f8nwc\") pod \"payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.131503 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.131446 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" podStartSLOduration=1.7576257229999999 podStartE2EDuration="2.131396864s" totalImagesPullingTime="373.771141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:24.127338671 +0000 UTC m=+788.158399901" watchObservedRunningTime="2026-09-11 18:07:24.131396864 +0000 UTC m=+788.162458097" Sep 11 18:07:24.188025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.187924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.188025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.187973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8nwc\" (UniqueName: \"kubernetes.io/projected/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-kube-api-access-f8nwc\") pod \"payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.188025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.188017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a481d95a-e6ab-4098-a823-1da5a8450937-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.188291 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.188061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2949d\" (UniqueName: \"kubernetes.io/projected/a481d95a-e6ab-4098-a823-1da5a8450937-kube-api-access-2949d\") pod \"payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.189024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.188990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.208068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.208043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8nwc\" (UniqueName: \"kubernetes.io/projected/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-kube-api-access-f8nwc\") pod \"payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.255662 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.255631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:24.289325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.289291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a481d95a-e6ab-4098-a823-1da5a8450937-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.289526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.289348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2949d\" (UniqueName: \"kubernetes.io/projected/a481d95a-e6ab-4098-a823-1da5a8450937-kube-api-access-2949d\") pod \"payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.290036 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.290013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a481d95a-e6ab-4098-a823-1da5a8450937-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.305343 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.305307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2949d\" (UniqueName: \"kubernetes.io/projected/a481d95a-e6ab-4098-a823-1da5a8450937-kube-api-access-2949d\") pod \"payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.329685 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.329652 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8"] Sep 11 18:07:24.331633 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.331608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:24.334472 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.334452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.341599 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.337933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-93598a-serving-cert\"" Sep 11 18:07:24.359801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.359771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8"] Sep 11 18:07:24.412716 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.412663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:24.491782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.491702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-metrics-certs\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.491782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.491752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-maas-api-tls\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.491980 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.491796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48cv\" (UniqueName: \"kubernetes.io/projected/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-kube-api-access-c48cv\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.491980 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.491821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-openshift-service-ca\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.536265 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.536236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:24.553620 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.553569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs"] Sep 11 18:07:24.554365 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:24.554342 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce42301_6248_4ad9_a6dd_d04a0f2ed67b.slice/crio-e4d5bb41c20727c2cbda62d13fcacbb3f3257b4cb6fd18e02747a734a87fe443 WatchSource:0}: Error finding container e4d5bb41c20727c2cbda62d13fcacbb3f3257b4cb6fd18e02747a734a87fe443: Status 404 returned error can't find the container with id e4d5bb41c20727c2cbda62d13fcacbb3f3257b4cb6fd18e02747a734a87fe443 Sep 11 18:07:24.592603 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.592575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-metrics-certs\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.592713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.592655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-maas-api-tls\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.592713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.592688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48cv\" (UniqueName: \"kubernetes.io/projected/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-kube-api-access-c48cv\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.592807 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.592727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-openshift-service-ca\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.593599 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.593577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-openshift-service-ca\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.595258 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.595236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-maas-api-tls\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.595523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.595502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-metrics-certs\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.602960 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.602938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48cv\" (UniqueName: \"kubernetes.io/projected/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-kube-api-access-c48cv\") pod \"maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.624224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.624200 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g"] Sep 11 18:07:24.625303 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:24.625280 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda481d95a_e6ab_4098_a823_1da5a8450937.slice/crio-b31d077919dbad41f7bb22b0cc383c8e579e40ebc91057e8c02aa22883ba1db9 WatchSource:0}: Error finding container b31d077919dbad41f7bb22b0cc383c8e579e40ebc91057e8c02aa22883ba1db9: Status 404 returned error can't find the container with id b31d077919dbad41f7bb22b0cc383c8e579e40ebc91057e8c02aa22883ba1db9 Sep 11 18:07:24.649581 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.649560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:24.665363 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.665339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:24.948913 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:24.948886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8"] Sep 11 18:07:24.950472 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:24.950441 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3558a1_e6d0_482c_9d1e_8bf5fb9e2142.slice/crio-bd166e0d3c112b803486a3399fbe3b05e4cc67acf6e33c736e1aa1383c01c0e6 WatchSource:0}: Error finding container bd166e0d3c112b803486a3399fbe3b05e4cc67acf6e33c736e1aa1383c01c0e6: Status 404 returned error can't find the container with id bd166e0d3c112b803486a3399fbe3b05e4cc67acf6e33c736e1aa1383c01c0e6 Sep 11 18:07:25.063953 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.063912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" event={"ID":"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b","Type":"ContainerStarted","Data":"789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0"} Sep 11 18:07:25.064150 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.063983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" event={"ID":"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b","Type":"ContainerStarted","Data":"e4d5bb41c20727c2cbda62d13fcacbb3f3257b4cb6fd18e02747a734a87fe443"} Sep 11 18:07:25.064832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.064778 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:25.068964 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.068917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" event={"ID":"a481d95a-e6ab-4098-a823-1da5a8450937","Type":"ContainerStarted","Data":"fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927"} Sep 11 18:07:25.069196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.069161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:25.069378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.069327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" event={"ID":"a481d95a-e6ab-4098-a823-1da5a8450937","Type":"ContainerStarted","Data":"b31d077919dbad41f7bb22b0cc383c8e579e40ebc91057e8c02aa22883ba1db9"} Sep 11 18:07:25.071103 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.071078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" event={"ID":"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142","Type":"ContainerStarted","Data":"bd166e0d3c112b803486a3399fbe3b05e4cc67acf6e33c736e1aa1383c01c0e6"} Sep 11 18:07:25.071447 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.071385 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:25.095841 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.095793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" podStartSLOduration=2.09577772 podStartE2EDuration="2.09577772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:25.094571773 +0000 UTC m=+789.125633031" watchObservedRunningTime="2026-09-11 18:07:25.09577772 +0000 UTC m=+789.126838949" Sep 11 18:07:25.123155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.123112 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" podStartSLOduration=2.123095057 podStartE2EDuration="2.123095057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:25.1196171 +0000 UTC m=+789.150678349" watchObservedRunningTime="2026-09-11 18:07:25.123095057 +0000 UTC m=+789.154156291" Sep 11 18:07:25.133583 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.133558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:25.362695 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.362565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:25.810316 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:25.810284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:26.078843 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:26.078744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" event={"ID":"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142","Type":"ContainerStarted","Data":"ca3e170491c4d02165443795566e527e50c9c80634c046b1de91a9e6d8a4ac4c"} Sep 11 18:07:26.079263 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:26.079033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:26.114563 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:26.114509 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" podStartSLOduration=1.797389057 podStartE2EDuration="2.114490979s" totalImagesPullingTime="317.101922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:26.112204457 +0000 UTC m=+790.143265719" watchObservedRunningTime="2026-09-11 18:07:26.114490979 +0000 UTC m=+790.145552255" Sep 11 18:07:26.849141 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:26.849106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:27.015363 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:27.015322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:27.129036 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:27.128987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:28.310134 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:28.310101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:29.058850 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:29.058813 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:07:29.585143 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:29.585106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:30.005387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.005359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:30.308869 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.308767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:30.447757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.447728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:30.544856 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.544821 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8"] Sep 11 18:07:30.548911 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.548890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.552383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.552358 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-736ba3-dockercfg-pvcvv\"" Sep 11 18:07:30.552513 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.552400 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-736ba3\"" Sep 11 18:07:30.570921 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.570837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8"] Sep 11 18:07:30.608076 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.608035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:30.657550 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.657507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d23de337-a2e9-4eed-8701-575f00f87636-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.657705 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.657622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4nl\" (UniqueName: \"kubernetes.io/projected/d23de337-a2e9-4eed-8701-575f00f87636-kube-api-access-rw4nl\") pod \"payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.759023 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.758977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4nl\" (UniqueName: \"kubernetes.io/projected/d23de337-a2e9-4eed-8701-575f00f87636-kube-api-access-rw4nl\") pod \"payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.759210 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.759125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d23de337-a2e9-4eed-8701-575f00f87636-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.760127 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.760092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d23de337-a2e9-4eed-8701-575f00f87636-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.779858 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.779821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4nl\" (UniqueName: \"kubernetes.io/projected/d23de337-a2e9-4eed-8701-575f00f87636-kube-api-access-rw4nl\") pod \"payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.861633 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.859314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:30.935117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:30.935083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:31.091439 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:31.091031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:07:31.126411 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:31.126386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:31.220784 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:31.220757 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8"] Sep 11 18:07:31.225222 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:31.225199 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23de337_a2e9_4eed_8701_575f00f87636.slice/crio-e058515d05bbf8c73b10b75a40c07943bf76ae16cabd0c08b25a579d2e6885c3 WatchSource:0}: Error finding container e058515d05bbf8c73b10b75a40c07943bf76ae16cabd0c08b25a579d2e6885c3: Status 404 returned error can't find the container with id e058515d05bbf8c73b10b75a40c07943bf76ae16cabd0c08b25a579d2e6885c3 Sep 11 18:07:31.539325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:31.539217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:32.091202 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:32.091176 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:07:32.113775 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:32.113747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" event={"ID":"d23de337-a2e9-4eed-8701-575f00f87636","Type":"ContainerStarted","Data":"cefa6a40ab0264f9f69891bcee4550d7d02030f152ee2c86c7756fd2fe7de315"} Sep 11 18:07:32.113928 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:32.113800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" event={"ID":"d23de337-a2e9-4eed-8701-575f00f87636","Type":"ContainerStarted","Data":"e058515d05bbf8c73b10b75a40c07943bf76ae16cabd0c08b25a579d2e6885c3"} Sep 11 18:07:32.113928 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:32.113849 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:32.142322 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:32.142266 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" podStartSLOduration=2.142244144 podStartE2EDuration="2.142244144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:32.140658063 +0000 UTC m=+796.171719307" watchObservedRunningTime="2026-09-11 18:07:32.142244144 +0000 UTC m=+796.173305376" Sep 11 18:07:32.250145 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:32.250111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:33.377715 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:33.377672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:34.043964 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:34.043928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:34.152871 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:34.152833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:34.321718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:34.321644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:34.637988 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:34.637960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:34.828410 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:34.828371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:35.035217 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:35.035136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:35.159117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:35.159079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:35.327766 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:35.327683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:35.447380 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:35.447348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:35.545924 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:35.545883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:36.081625 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:36.081599 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:07:36.347905 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:36.347828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:37.087985 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:37.087956 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:07:37.352437 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:37.352344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:37.707362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:37.707297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:38.148729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.148690 2579 generic.go:356] "Generic (PLEG): container finished" podID="65873c38-8f3d-4261-972e-23c9b1d91664" containerID="1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725" exitCode=0 Sep 11 18:07:38.149093 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.148773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" event={"ID":"65873c38-8f3d-4261-972e-23c9b1d91664","Type":"ContainerDied","Data":"1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725"} Sep 11 18:07:38.149202 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.149187 2579 scope.go:117] "RemoveContainer" containerID="1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725" Sep 11 18:07:38.150458 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.150408 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:07:38.150560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.150506 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:07:38.150560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.150557 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:07:38.486812 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:38.486711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:39.156513 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.156481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" event={"ID":"65873c38-8f3d-4261-972e-23c9b1d91664","Type":"ContainerStarted","Data":"17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd"} Sep 11 18:07:39.466509 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.464249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:39.570023 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.569991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:39.751876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.751789 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:39.752024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.751987 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 11 18:07:39.752077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.752047 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 11 18:07:39.925971 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:39.925934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:40.070256 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.070173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:40.226830 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.226800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:40.357107 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.357017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:40.475481 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.475402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:40.532152 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.532118 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl"] Sep 11 18:07:40.534619 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.534604 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.538152 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.538127 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-33958e97-dockercfg-rk82n\"" Sep 11 18:07:40.538517 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.538495 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-33958e97\"" Sep 11 18:07:40.551124 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.551101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl"] Sep 11 18:07:40.590827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.588317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdsss\" (UniqueName: \"kubernetes.io/projected/456ce7fe-fa59-4b37-904f-63dc07e19d5f-kube-api-access-tdsss\") pod \"payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.591047 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.591009 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx"] Sep 11 18:07:40.591246 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.591223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456ce7fe-fa59-4b37-904f-63dc07e19d5f-plugins-config-volume\") pod \"payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.594704 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.594677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.603175 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.603151 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx"] Sep 11 18:07:40.605455 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.605354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:40.697030 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.696975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456ce7fe-fa59-4b37-904f-63dc07e19d5f-plugins-config-volume\") pod \"payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.697239 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.697075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ad465-8730-43cc-83c7-c459e03b58de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.697239 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.697230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fg8p\" (UniqueName: \"kubernetes.io/projected/250ad465-8730-43cc-83c7-c459e03b58de-kube-api-access-2fg8p\") pod \"payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.697364 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.697312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdsss\" (UniqueName: \"kubernetes.io/projected/456ce7fe-fa59-4b37-904f-63dc07e19d5f-kube-api-access-tdsss\") pod \"payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.698616 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.698588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456ce7fe-fa59-4b37-904f-63dc07e19d5f-plugins-config-volume\") pod \"payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.706929 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.706906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdsss\" (UniqueName: \"kubernetes.io/projected/456ce7fe-fa59-4b37-904f-63dc07e19d5f-kube-api-access-tdsss\") pod \"payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.751742 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.751713 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 11 18:07:40.751880 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.751766 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 11 18:07:40.798582 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.798536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ad465-8730-43cc-83c7-c459e03b58de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.798798 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.798607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fg8p\" (UniqueName: \"kubernetes.io/projected/250ad465-8730-43cc-83c7-c459e03b58de-kube-api-access-2fg8p\") pod \"payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.799275 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.799254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ad465-8730-43cc-83c7-c459e03b58de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.816176 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.816148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fg8p\" (UniqueName: \"kubernetes.io/projected/250ad465-8730-43cc-83c7-c459e03b58de-kube-api-access-2fg8p\") pod \"payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:40.846104 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.846065 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:40.908506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:40.908477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:41.146371 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.146333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:41.156191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.156165 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl"] Sep 11 18:07:41.158227 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:41.158199 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456ce7fe_fa59_4b37_904f_63dc07e19d5f.slice/crio-4d3dadeff6d875f5874d206674eefc790fd9fe5018e856cf22fdc318bb8ee3f3 WatchSource:0}: Error finding container 4d3dadeff6d875f5874d206674eefc790fd9fe5018e856cf22fdc318bb8ee3f3: Status 404 returned error can't find the container with id 4d3dadeff6d875f5874d206674eefc790fd9fe5018e856cf22fdc318bb8ee3f3 Sep 11 18:07:41.182296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.182202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" event={"ID":"456ce7fe-fa59-4b37-904f-63dc07e19d5f","Type":"ContainerStarted","Data":"4d3dadeff6d875f5874d206674eefc790fd9fe5018e856cf22fdc318bb8ee3f3"} Sep 11 18:07:41.224077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.224048 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx"] Sep 11 18:07:41.225028 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:41.225005 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250ad465_8730_43cc_83c7_c459e03b58de.slice/crio-f70bdec1c9073ef5f18eb632eaee78baaf660f8a9de55bdc317cee62a1e65068 WatchSource:0}: Error finding container f70bdec1c9073ef5f18eb632eaee78baaf660f8a9de55bdc317cee62a1e65068: Status 404 returned error can't find the container with id f70bdec1c9073ef5f18eb632eaee78baaf660f8a9de55bdc317cee62a1e65068 Sep 11 18:07:41.396409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.396376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:41.751769 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.751737 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 11 18:07:41.751943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:41.751790 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 11 18:07:42.165967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.165934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:42.196471 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.196395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" event={"ID":"250ad465-8730-43cc-83c7-c459e03b58de","Type":"ContainerStarted","Data":"8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e"} Sep 11 18:07:42.196789 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.196767 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" event={"ID":"250ad465-8730-43cc-83c7-c459e03b58de","Type":"ContainerStarted","Data":"f70bdec1c9073ef5f18eb632eaee78baaf660f8a9de55bdc317cee62a1e65068"} Sep 11 18:07:42.198014 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.197846 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:07:42.198773 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.198753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" event={"ID":"456ce7fe-fa59-4b37-904f-63dc07e19d5f","Type":"ContainerStarted","Data":"bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c"} Sep 11 18:07:42.199011 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.199001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:42.221897 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.221841 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" podStartSLOduration=2.221823208 podStartE2EDuration="2.221823208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:42.220606502 +0000 UTC m=+806.251667770" watchObservedRunningTime="2026-09-11 18:07:42.221823208 +0000 UTC m=+806.252884439" Sep 11 18:07:42.242776 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.242719 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" podStartSLOduration=2.242703634 podStartE2EDuration="2.242703634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:42.240198719 +0000 UTC m=+806.271260050" watchObservedRunningTime="2026-09-11 18:07:42.242703634 +0000 UTC m=+806.273764865" Sep 11 18:07:42.751485 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.751444 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 11 18:07:42.751937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:42.751511 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 11 18:07:43.121481 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:43.121452 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:07:43.539382 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:43.539263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:43.696380 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:43.696330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:43.751466 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:43.751406 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Sep 11 18:07:43.751662 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:43.751497 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Sep 11 18:07:43.804526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:43.804411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:44.720623 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:44.720592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:44.760506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:44.760476 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:44.760937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:44.760833 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:44.761533 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:44.761509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:07:45.718675 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:45.718638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:46.638353 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:46.638319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:47.463053 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:47.463025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:47.627914 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:47.627870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:48.127555 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.127505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:48.259050 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.259011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:48.385855 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.385767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:48.513305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.513272 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9"] Sep 11 18:07:48.516239 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.516219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.520527 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.520494 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d59a02-dockercfg-jvc4s\"" Sep 11 18:07:48.520674 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.520534 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d59a02\"" Sep 11 18:07:48.548293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.548262 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9"] Sep 11 18:07:48.578625 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.578596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqc4\" (UniqueName: \"kubernetes.io/projected/cf9b1b18-2540-459d-8807-32511ebb2d5d-kube-api-access-jbqc4\") pod \"payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.578795 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.578761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b1b18-2540-459d-8807-32511ebb2d5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.679999 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.679921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b1b18-2540-459d-8807-32511ebb2d5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.679999 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.679967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqc4\" (UniqueName: \"kubernetes.io/projected/cf9b1b18-2540-459d-8807-32511ebb2d5d-kube-api-access-jbqc4\") pod \"payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.680647 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.680629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b1b18-2540-459d-8807-32511ebb2d5d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.691313 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.691277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqc4\" (UniqueName: \"kubernetes.io/projected/cf9b1b18-2540-459d-8807-32511ebb2d5d-kube-api-access-jbqc4\") pod \"payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:48.747805 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.747769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:48.827495 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:48.827463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:49.107045 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:49.107018 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9"] Sep 11 18:07:49.109037 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:07:49.109011 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9b1b18_2540_459d_8807_32511ebb2d5d.slice/crio-1c30ccd6ceb4203562a064b8c7103af2461ab505dfe23f1986051cc19354b1de WatchSource:0}: Error finding container 1c30ccd6ceb4203562a064b8c7103af2461ab505dfe23f1986051cc19354b1de: Status 404 returned error can't find the container with id 1c30ccd6ceb4203562a064b8c7103af2461ab505dfe23f1986051cc19354b1de Sep 11 18:07:49.242002 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:49.241966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" event={"ID":"cf9b1b18-2540-459d-8807-32511ebb2d5d","Type":"ContainerStarted","Data":"9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3"} Sep 11 18:07:49.242392 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:49.242030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" event={"ID":"cf9b1b18-2540-459d-8807-32511ebb2d5d","Type":"ContainerStarted","Data":"1c30ccd6ceb4203562a064b8c7103af2461ab505dfe23f1986051cc19354b1de"} Sep 11 18:07:49.242392 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:49.242144 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:07:49.289267 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:49.289149 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" podStartSLOduration=1.2891300669999999 podStartE2EDuration="1.289130067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:49.288307768 +0000 UTC m=+813.319369020" watchObservedRunningTime="2026-09-11 18:07:49.289130067 +0000 UTC m=+813.320191299" Sep 11 18:07:49.772002 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:49.771966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:50.611746 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:50.611714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:50.836094 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:50.836057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:51.397228 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:51.397195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:52.465879 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:52.465842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:53.207568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:53.207536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:07:53.532372 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:53.532282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:07:54.213486 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:07:54.213455 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:08:00.249993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:00.249962 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:08:00.378405 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:00.378357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:00.484293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:00.484259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:00.641021 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:00.640987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:01.062579 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:01.062486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:02.878073 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:02.878037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:03.705464 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:03.705405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:04.566961 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:04.566931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:05.514832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:05.514801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:06.300372 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:06.300340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:07.095124 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:07.095094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:07.346818 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:07.346732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:08.444039 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:08.444005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:08.664947 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:08.664914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:09.145475 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:09.145432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:09.300522 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:09.300482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:09.421393 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:09.421316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:09.583370 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:09.583340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:10.410815 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:10.410777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:11.218814 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:11.218773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:15.717891 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:15.717858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:16.971156 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:16.971121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:17.011852 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.011805 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl"] Sep 11 18:08:17.012080 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.012031 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" podUID="456ce7fe-fa59-4b37-904f-63dc07e19d5f" containerName="payload-processing" containerID="cri-o://bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c" gracePeriod=30 Sep 11 18:08:17.030043 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.030009 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx"] Sep 11 18:08:17.030270 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.030246 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" podUID="250ad465-8730-43cc-83c7-c459e03b58de" containerName="payload-pre-processing" containerID="cri-o://8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e" gracePeriod=30 Sep 11 18:08:17.161771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.161734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:17.292155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.292129 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:08:17.321685 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.321662 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:08:17.380375 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.380343 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdsss\" (UniqueName: \"kubernetes.io/projected/456ce7fe-fa59-4b37-904f-63dc07e19d5f-kube-api-access-tdsss\") pod \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " Sep 11 18:08:17.380585 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.380498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456ce7fe-fa59-4b37-904f-63dc07e19d5f-plugins-config-volume\") pod \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\" (UID: \"456ce7fe-fa59-4b37-904f-63dc07e19d5f\") " Sep 11 18:08:17.380910 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.380872 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456ce7fe-fa59-4b37-904f-63dc07e19d5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "456ce7fe-fa59-4b37-904f-63dc07e19d5f" (UID: "456ce7fe-fa59-4b37-904f-63dc07e19d5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:08:17.381391 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.381087 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fg8p\" (UniqueName: \"kubernetes.io/projected/250ad465-8730-43cc-83c7-c459e03b58de-kube-api-access-2fg8p\") pod \"250ad465-8730-43cc-83c7-c459e03b58de\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " Sep 11 18:08:17.381391 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.381137 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ad465-8730-43cc-83c7-c459e03b58de-plugins-config-volume\") pod \"250ad465-8730-43cc-83c7-c459e03b58de\" (UID: \"250ad465-8730-43cc-83c7-c459e03b58de\") " Sep 11 18:08:17.381620 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.381534 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456ce7fe-fa59-4b37-904f-63dc07e19d5f-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:08:17.381913 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.381885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250ad465-8730-43cc-83c7-c459e03b58de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "250ad465-8730-43cc-83c7-c459e03b58de" (UID: "250ad465-8730-43cc-83c7-c459e03b58de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:08:17.384110 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.383524 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456ce7fe-fa59-4b37-904f-63dc07e19d5f-kube-api-access-tdsss" (OuterVolumeSpecName: "kube-api-access-tdsss") pod "456ce7fe-fa59-4b37-904f-63dc07e19d5f" (UID: "456ce7fe-fa59-4b37-904f-63dc07e19d5f"). InnerVolumeSpecName "kube-api-access-tdsss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:08:17.391157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.391106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250ad465-8730-43cc-83c7-c459e03b58de-kube-api-access-2fg8p" (OuterVolumeSpecName: "kube-api-access-2fg8p") pod "250ad465-8730-43cc-83c7-c459e03b58de" (UID: "250ad465-8730-43cc-83c7-c459e03b58de"). InnerVolumeSpecName "kube-api-access-2fg8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:08:17.393531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.393493 2579 generic.go:356] "Generic (PLEG): container finished" podID="250ad465-8730-43cc-83c7-c459e03b58de" containerID="8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e" exitCode=0 Sep 11 18:08:17.393642 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.393598 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" Sep 11 18:08:17.393642 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.393615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" event={"ID":"250ad465-8730-43cc-83c7-c459e03b58de","Type":"ContainerDied","Data":"8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e"} Sep 11 18:08:17.393735 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.393703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx" event={"ID":"250ad465-8730-43cc-83c7-c459e03b58de","Type":"ContainerDied","Data":"f70bdec1c9073ef5f18eb632eaee78baaf660f8a9de55bdc317cee62a1e65068"} Sep 11 18:08:17.393782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.393737 2579 scope.go:117] "RemoveContainer" containerID="8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e" Sep 11 18:08:17.395825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.395795 2579 generic.go:356] "Generic (PLEG): container finished" podID="456ce7fe-fa59-4b37-904f-63dc07e19d5f" containerID="bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c" exitCode=0 Sep 11 18:08:17.395993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.395938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" event={"ID":"456ce7fe-fa59-4b37-904f-63dc07e19d5f","Type":"ContainerDied","Data":"bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c"} Sep 11 18:08:17.395993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.395985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" event={"ID":"456ce7fe-fa59-4b37-904f-63dc07e19d5f","Type":"ContainerDied","Data":"4d3dadeff6d875f5874d206674eefc790fd9fe5018e856cf22fdc318bb8ee3f3"} Sep 11 18:08:17.396119 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.396065 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl" Sep 11 18:08:17.418986 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.418967 2579 scope.go:117] "RemoveContainer" containerID="8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e" Sep 11 18:08:17.419394 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:08:17.419374 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e\": container with ID starting with 8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e not found: ID does not exist" containerID="8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e" Sep 11 18:08:17.419467 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.419406 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e"} err="failed to get container status \"8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e\": rpc error: code = NotFound desc = could not find container \"8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e\": container with ID starting with 8eb21b8f99973fd68f4ef0b0761d2ec1046ba7d4652b9e4b99b9eb6f10c1262e not found: ID does not exist" Sep 11 18:08:17.419467 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.419455 2579 scope.go:117] "RemoveContainer" containerID="bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c" Sep 11 18:08:17.430055 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.430039 2579 scope.go:117] "RemoveContainer" containerID="bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c" Sep 11 18:08:17.430528 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:08:17.430500 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c\": container with ID starting with bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c not found: ID does not exist" containerID="bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c" Sep 11 18:08:17.430614 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.430535 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c"} err="failed to get container status \"bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c\": rpc error: code = NotFound desc = could not find container \"bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c\": container with ID starting with bcfdafffc7b9dca56a6b7e762ef457a238612d4e32b301328f9dd339eee43a1c not found: ID does not exist" Sep 11 18:08:17.443840 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.443813 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx"] Sep 11 18:08:17.466180 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.466147 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33958e97-5c79855854-75bfx"] Sep 11 18:08:17.484014 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.483932 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fg8p\" (UniqueName: \"kubernetes.io/projected/250ad465-8730-43cc-83c7-c459e03b58de-kube-api-access-2fg8p\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:08:17.484014 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.483957 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250ad465-8730-43cc-83c7-c459e03b58de-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:08:17.484014 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.483972 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdsss\" (UniqueName: \"kubernetes.io/projected/456ce7fe-fa59-4b37-904f-63dc07e19d5f-kube-api-access-tdsss\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:08:17.485452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.485403 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl"] Sep 11 18:08:17.495068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.495044 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33958e97-5744fc4cbf-jcxvl"] Sep 11 18:08:17.569578 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.569543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:17.806194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:17.806104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:18.171868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.171831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:18.423918 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.423837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:18.444429 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.444383 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250ad465-8730-43cc-83c7-c459e03b58de" path="/var/lib/kubelet/pods/250ad465-8730-43cc-83c7-c459e03b58de/volumes" Sep 11 18:08:18.444811 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.444796 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="456ce7fe-fa59-4b37-904f-63dc07e19d5f" path="/var/lib/kubelet/pods/456ce7fe-fa59-4b37-904f-63dc07e19d5f/volumes" Sep 11 18:08:18.573700 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.573669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:18.953197 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.953161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:18.971181 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971145 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw"] Sep 11 18:08:18.971752 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971731 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456ce7fe-fa59-4b37-904f-63dc07e19d5f" containerName="payload-processing" Sep 11 18:08:18.971752 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971751 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="456ce7fe-fa59-4b37-904f-63dc07e19d5f" containerName="payload-processing" Sep 11 18:08:18.971910 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971760 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250ad465-8730-43cc-83c7-c459e03b58de" containerName="payload-pre-processing" Sep 11 18:08:18.971910 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971765 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="250ad465-8730-43cc-83c7-c459e03b58de" containerName="payload-pre-processing" Sep 11 18:08:18.971910 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971829 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="456ce7fe-fa59-4b37-904f-63dc07e19d5f" containerName="payload-processing" Sep 11 18:08:18.971910 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.971835 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="250ad465-8730-43cc-83c7-c459e03b58de" containerName="payload-pre-processing" Sep 11 18:08:18.977038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.977021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:18.981819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:18.981797 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-44ba3006-gw-openshift-default-dockercfg-v24gf\"" Sep 11 18:08:19.003315 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.003281 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw"] Sep 11 18:08:19.099909 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.099879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-socket\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100108 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.099918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-certs\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100108 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.099939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-data\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100108 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.100086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-token\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.100169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb78q\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-kube-api-access-sb78q\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.100216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-envoy\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.100243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istiod-ca-cert\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.100274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-podinfo\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.100509 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.100371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-credential-socket\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.182902 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.182867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:19.201172 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-credential-socket\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-socket\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-certs\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-data\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-token\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb78q\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-kube-api-access-sb78q\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-envoy\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istiod-ca-cert\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-podinfo\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.201622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.201607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-credential-socket\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.202216 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.202186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-socket\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.202483 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.202460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-certs\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.202736 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.202712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-data\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.203124 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.203096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istiod-ca-cert\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.207566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.206129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-podinfo\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.207695 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.207587 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-envoy\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.217399 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.217374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-token\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.218501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.218484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb78q\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-kube-api-access-sb78q\") pod \"e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.293875 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.293841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:19.583698 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.583671 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw"] Sep 11 18:08:19.584872 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:08:19.584847 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85c8eae_f9f2_4e08_9d39_7dabd95c7e3d.slice/crio-d62f1db2ec9c658ff23c85c49a0d0bdb9871e64d96642c10dc0395e2634974ef WatchSource:0}: Error finding container d62f1db2ec9c658ff23c85c49a0d0bdb9871e64d96642c10dc0395e2634974ef: Status 404 returned error can't find the container with id d62f1db2ec9c658ff23c85c49a0d0bdb9871e64d96642c10dc0395e2634974ef Sep 11 18:08:19.587429 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.587384 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:08:19.587505 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.587474 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:08:19.587544 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.587505 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:08:19.781254 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:19.781220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:20.416693 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:20.416652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" event={"ID":"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d","Type":"ContainerStarted","Data":"302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f"} Sep 11 18:08:20.417081 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:20.416706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" event={"ID":"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d","Type":"ContainerStarted","Data":"d62f1db2ec9c658ff23c85c49a0d0bdb9871e64d96642c10dc0395e2634974ef"} Sep 11 18:08:20.424930 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:20.424901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:20.451898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:20.451849 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podStartSLOduration=2.451832343 podStartE2EDuration="2.451832343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:20.449333962 +0000 UTC m=+844.480395205" watchObservedRunningTime="2026-09-11 18:08:20.451832343 +0000 UTC m=+844.482893572" Sep 11 18:08:20.559004 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:20.558970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:20.813686 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:20.813599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:21.294503 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:21.294471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:21.300815 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:21.300791 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:21.421698 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:21.421660 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:21.422673 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:21.422658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:21.695572 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:21.695537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:21.955010 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:21.954921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:22.088249 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:22.088211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:22.718067 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:22.718034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:23.659039 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:23.659008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:24.603207 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:24.603172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:24.701694 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:24.701658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:25.077890 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.077858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:25.320477 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.320442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:25.469151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.469120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:25.551457 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.551407 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56"] Sep 11 18:08:25.556120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.556104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.559596 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.559572 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-44ba3006-dockercfg-rtwwl\"" Sep 11 18:08:25.559715 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.559610 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-44ba3006\"" Sep 11 18:08:25.567827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.567791 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56"] Sep 11 18:08:25.669911 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.669878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/0a2f5e48-6191-4f7e-9164-5e6e307a8389-kube-api-access-6v9h9\") pod \"payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.670301 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.669971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f5e48-6191-4f7e-9164-5e6e307a8389-plugins-config-volume\") pod \"payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.770841 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.770739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/0a2f5e48-6191-4f7e-9164-5e6e307a8389-kube-api-access-6v9h9\") pod \"payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.770841 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.770796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f5e48-6191-4f7e-9164-5e6e307a8389-plugins-config-volume\") pod \"payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.771591 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.771568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f5e48-6191-4f7e-9164-5e6e307a8389-plugins-config-volume\") pod \"payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.780471 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.780441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/0a2f5e48-6191-4f7e-9164-5e6e307a8389-kube-api-access-6v9h9\") pod \"payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.868166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.868133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:25.929270 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:25.928758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:26.145625 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:26.145599 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56"] Sep 11 18:08:26.147061 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:08:26.147034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2f5e48_6191_4f7e_9164_5e6e307a8389.slice/crio-1de59d1de9c2736a0b42e6ba914abbed6359d82a18879a32e65e9d1cc3dbf7c6 WatchSource:0}: Error finding container 1de59d1de9c2736a0b42e6ba914abbed6359d82a18879a32e65e9d1cc3dbf7c6: Status 404 returned error can't find the container with id 1de59d1de9c2736a0b42e6ba914abbed6359d82a18879a32e65e9d1cc3dbf7c6 Sep 11 18:08:26.455562 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:26.455464 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" event={"ID":"0a2f5e48-6191-4f7e-9164-5e6e307a8389","Type":"ContainerStarted","Data":"317b6c120c901d093bf7f23a30487a55c8c3027d507809011a32b688f1d3832e"} Sep 11 18:08:26.455562 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:26.455528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" event={"ID":"0a2f5e48-6191-4f7e-9164-5e6e307a8389","Type":"ContainerStarted","Data":"1de59d1de9c2736a0b42e6ba914abbed6359d82a18879a32e65e9d1cc3dbf7c6"} Sep 11 18:08:26.455562 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:26.455557 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:26.477516 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:26.477469 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" podStartSLOduration=1.477452531 podStartE2EDuration="1.477452531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:26.475180827 +0000 UTC m=+850.506242074" watchObservedRunningTime="2026-09-11 18:08:26.477452531 +0000 UTC m=+850.508513758" Sep 11 18:08:26.759620 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:26.759534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:27.580880 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:27.580846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:28.445084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:28.445008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:33.022884 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:33.022851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:37.463626 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:37.463593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:08:44.547046 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:44.547012 2579 generic.go:356] "Generic (PLEG): container finished" podID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerID="302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f" exitCode=0 Sep 11 18:08:44.547508 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:44.547088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" event={"ID":"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d","Type":"ContainerDied","Data":"302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f"} Sep 11 18:08:44.547566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:44.547540 2579 scope.go:117] "RemoveContainer" containerID="302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f" Sep 11 18:08:44.548768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:44.548744 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:08:44.548826 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:44.548807 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:08:44.548865 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:44.548837 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:08:45.554205 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:45.554168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" event={"ID":"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d","Type":"ContainerStarted","Data":"acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926"} Sep 11 18:08:46.294506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:46.294470 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:46.294750 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:46.294729 2579 patch_prober.go:28] interesting pod/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 11 18:08:46.294837 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:46.294776 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 11 18:08:47.294757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:47.294723 2579 patch_prober.go:28] interesting pod/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 11 18:08:47.295132 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:47.294777 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 11 18:08:47.640193 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:47.640161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:48.294636 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:48.294574 2579 patch_prober.go:28] interesting pod/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 11 18:08:48.294884 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:48.294678 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 11 18:08:48.803385 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:48.803350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:49.294309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:49.294269 2579 patch_prober.go:28] interesting pod/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 11 18:08:49.294594 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:49.294328 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 11 18:08:50.136658 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:50.136614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:50.294828 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:50.294792 2579 patch_prober.go:28] interesting pod/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 11 18:08:50.295042 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:50.294858 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 11 18:08:51.029827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:51.029790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:51.298852 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:51.298770 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:51.299291 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:51.299146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:51.299870 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:51.299849 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:08:52.153365 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:52.153336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:53.135473 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:53.135410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:54.010564 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:54.010531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:54.796177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:54.796118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:55.803938 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:55.803845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:56.743690 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:56.743629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:56.872268 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:56.872233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:57.680872 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:57.680835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:08:58.517385 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:08:58.517353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:03.236374 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:03.236341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:04.366384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.366337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:04.405979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.405948 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56"] Sep 11 18:09:04.406195 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.406170 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" podUID="0a2f5e48-6191-4f7e-9164-5e6e307a8389" containerName="payload-processing" containerID="cri-o://317b6c120c901d093bf7f23a30487a55c8c3027d507809011a32b688f1d3832e" gracePeriod=30 Sep 11 18:09:04.480585 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.480555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:04.618390 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.618297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:04.664835 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.664792 2579 generic.go:356] "Generic (PLEG): container finished" podID="0a2f5e48-6191-4f7e-9164-5e6e307a8389" containerID="317b6c120c901d093bf7f23a30487a55c8c3027d507809011a32b688f1d3832e" exitCode=0 Sep 11 18:09:04.664994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.664860 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" event={"ID":"0a2f5e48-6191-4f7e-9164-5e6e307a8389","Type":"ContainerDied","Data":"317b6c120c901d093bf7f23a30487a55c8c3027d507809011a32b688f1d3832e"} Sep 11 18:09:04.664994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.664940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" event={"ID":"0a2f5e48-6191-4f7e-9164-5e6e307a8389","Type":"ContainerDied","Data":"1de59d1de9c2736a0b42e6ba914abbed6359d82a18879a32e65e9d1cc3dbf7c6"} Sep 11 18:09:04.664994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.664966 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de59d1de9c2736a0b42e6ba914abbed6359d82a18879a32e65e9d1cc3dbf7c6" Sep 11 18:09:04.674755 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.674729 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:09:04.731697 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.731647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:04.777308 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.777279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/0a2f5e48-6191-4f7e-9164-5e6e307a8389-kube-api-access-6v9h9\") pod \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " Sep 11 18:09:04.777514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.777361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f5e48-6191-4f7e-9164-5e6e307a8389-plugins-config-volume\") pod \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\" (UID: \"0a2f5e48-6191-4f7e-9164-5e6e307a8389\") " Sep 11 18:09:04.777777 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.777756 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2f5e48-6191-4f7e-9164-5e6e307a8389-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a2f5e48-6191-4f7e-9164-5e6e307a8389" (UID: "0a2f5e48-6191-4f7e-9164-5e6e307a8389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:09:04.779655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.779636 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2f5e48-6191-4f7e-9164-5e6e307a8389-kube-api-access-6v9h9" (OuterVolumeSpecName: "kube-api-access-6v9h9") pod "0a2f5e48-6191-4f7e-9164-5e6e307a8389" (UID: "0a2f5e48-6191-4f7e-9164-5e6e307a8389"). InnerVolumeSpecName "kube-api-access-6v9h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:09:04.859960 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.859925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:04.879234 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.879151 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2f5e48-6191-4f7e-9164-5e6e307a8389-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:04.879234 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.879187 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/0a2f5e48-6191-4f7e-9164-5e6e307a8389-kube-api-access-6v9h9\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:04.912387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.912353 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw"] Sep 11 18:09:04.912625 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.912601 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" containerID="cri-o://acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926" gracePeriod=30 Sep 11 18:09:04.976725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:04.976695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:05.370250 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:05.370221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:05.603895 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:05.603868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:05.669562 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:05.669461 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56" Sep 11 18:09:05.718791 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:05.718758 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56"] Sep 11 18:09:05.725323 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:05.725296 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-44ba3006-77c7c8b56b-gzf56"] Sep 11 18:09:05.899876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:05.899837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:06.077161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:06.077070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:06.302166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:06.302128 2579 patch_prober.go:28] interesting pod/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 11 18:09:06.302340 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:06.302195 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:09:06.444739 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:06.444705 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2f5e48-6191-4f7e-9164-5e6e307a8389" path="/var/lib/kubelet/pods/0a2f5e48-6191-4f7e-9164-5e6e307a8389/volumes" Sep 11 18:09:06.784300 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:06.784204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:08.255267 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:08.255151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:08.847430 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:08.847386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:08.988795 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:08.988749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:09.690778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:09.690726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:10.170651 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.170629 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:09:10.331414 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331321 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-token\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331414 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331371 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-data\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331414 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-envoy\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-certs\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb78q\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-kube-api-access-sb78q\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331581 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-socket\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331629 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istiod-ca-cert\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331689 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-podinfo\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.331725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:09:10.332031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331732 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-credential-socket\") pod \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\" (UID: \"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d\") " Sep 11 18:09:10.332031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331870 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:09:10.332031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.331882 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:09:10.332193 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.332090 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:09:10.332193 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.332109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:09:10.332193 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.332140 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.332305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.332192 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.332352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.332300 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.333857 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.333827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:09:10.333857 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.333841 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:09:10.334012 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.333957 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-kube-api-access-sb78q" (OuterVolumeSpecName: "kube-api-access-sb78q") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "kube-api-access-sb78q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:09:10.334127 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.334106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" (UID: "d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:09:10.433510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.433479 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.433510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.433507 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb78q\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-kube-api-access-sb78q\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.433510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.433517 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.433750 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.433534 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.433750 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.433543 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.433750 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.433551 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:10.698432 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.697162 2579 generic.go:356] "Generic (PLEG): container finished" podID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerID="acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926" exitCode=0 Sep 11 18:09:10.698432 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.697274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" event={"ID":"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d","Type":"ContainerDied","Data":"acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926"} Sep 11 18:09:10.698432 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.697323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" event={"ID":"d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d","Type":"ContainerDied","Data":"d62f1db2ec9c658ff23c85c49a0d0bdb9871e64d96642c10dc0395e2634974ef"} Sep 11 18:09:10.698432 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.697350 2579 scope.go:117] "RemoveContainer" containerID="acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926" Sep 11 18:09:10.698432 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.697559 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw" Sep 11 18:09:10.717832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.717790 2579 scope.go:117] "RemoveContainer" containerID="302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f" Sep 11 18:09:10.727271 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.727201 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw"] Sep 11 18:09:10.732438 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.731779 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-44ba3006-gw-openshift-default-9857697b4-ltdrw"] Sep 11 18:09:10.736560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.734672 2579 scope.go:117] "RemoveContainer" containerID="acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926" Sep 11 18:09:10.736560 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:09:10.735022 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926\": container with ID starting with acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926 not found: ID does not exist" containerID="acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926" Sep 11 18:09:10.736560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.735053 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926"} err="failed to get container status \"acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926\": rpc error: code = NotFound desc = could not find container \"acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926\": container with ID starting with acaeca1aab262fd8d54d688de8ae6b8c57427d4f21af8946da24be91e1e0d926 not found: ID does not exist" Sep 11 18:09:10.736560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.735078 2579 scope.go:117] "RemoveContainer" containerID="302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f" Sep 11 18:09:10.736560 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:09:10.735317 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f\": container with ID starting with 302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f not found: ID does not exist" containerID="302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f" Sep 11 18:09:10.736560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.735341 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f"} err="failed to get container status \"302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f\": rpc error: code = NotFound desc = could not find container \"302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f\": container with ID starting with 302c5653868d989ea481dfe8dde86bc9454a302d8f9ea44ee06852e3e838f74f not found: ID does not exist" Sep 11 18:09:10.992964 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:10.992846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:11.837387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:11.837351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:12.444658 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:12.444621 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" path="/var/lib/kubelet/pods/d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d/volumes" Sep 11 18:09:12.921968 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:12.921935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:13.043397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.043363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:13.206189 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.206105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:13.595854 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.595761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:13.676312 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676279 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm"] Sep 11 18:09:13.676722 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676707 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" Sep 11 18:09:13.676722 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676723 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" Sep 11 18:09:13.676825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676734 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2f5e48-6191-4f7e-9164-5e6e307a8389" containerName="payload-processing" Sep 11 18:09:13.676825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676740 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2f5e48-6191-4f7e-9164-5e6e307a8389" containerName="payload-processing" Sep 11 18:09:13.676825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676748 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" Sep 11 18:09:13.676825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676753 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" Sep 11 18:09:13.676967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676829 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2f5e48-6191-4f7e-9164-5e6e307a8389" containerName="payload-processing" Sep 11 18:09:13.676967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676839 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" Sep 11 18:09:13.676967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.676847 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85c8eae-f9f2-4e08-9d39-7dabd95c7e3d" containerName="istio-proxy" Sep 11 18:09:13.679848 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.679833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.683997 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.683968 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-820e15b2-dockercfg-88lvg\"" Sep 11 18:09:13.684134 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.684085 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-820e15b2\"" Sep 11 18:09:13.689911 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.689881 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx"] Sep 11 18:09:13.693247 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.693226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.699837 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.699813 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm"] Sep 11 18:09:13.710124 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.710101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx"] Sep 11 18:09:13.768413 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.768381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cv9\" (UniqueName: \"kubernetes.io/projected/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-kube-api-access-v5cv9\") pod \"payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.768587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.768505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-plugins-config-volume\") pod \"payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.815180 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.815149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:13.869297 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.869268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde58723-a523-4089-a206-d127438341e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.869476 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.869329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57gq\" (UniqueName: \"kubernetes.io/projected/dde58723-a523-4089-a206-d127438341e7-kube-api-access-d57gq\") pod \"payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.869476 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.869379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cv9\" (UniqueName: \"kubernetes.io/projected/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-kube-api-access-v5cv9\") pod \"payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.869578 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.869495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-plugins-config-volume\") pod \"payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.870187 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.870169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-plugins-config-volume\") pod \"payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.882384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.882361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cv9\" (UniqueName: \"kubernetes.io/projected/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-kube-api-access-v5cv9\") pod \"payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:13.970813 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.970780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde58723-a523-4089-a206-d127438341e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.971169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.970833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57gq\" (UniqueName: \"kubernetes.io/projected/dde58723-a523-4089-a206-d127438341e7-kube-api-access-d57gq\") pod \"payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.971491 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.971471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde58723-a523-4089-a206-d127438341e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.980849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.980810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57gq\" (UniqueName: \"kubernetes.io/projected/dde58723-a523-4089-a206-d127438341e7-kube-api-access-d57gq\") pod \"payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:13.990254 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:13.990234 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:14.007959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.007938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:14.140936 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.140850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:14.274055 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.274028 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm"] Sep 11 18:09:14.276252 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:09:14.276225 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa7e02e_6e6b_4418_84f3_7917ce7d4897.slice/crio-b376a48053189cb9bc437c4e2c9083895ebb85b0376e092263bd7883f0f3fc69 WatchSource:0}: Error finding container b376a48053189cb9bc437c4e2c9083895ebb85b0376e092263bd7883f0f3fc69: Status 404 returned error can't find the container with id b376a48053189cb9bc437c4e2c9083895ebb85b0376e092263bd7883f0f3fc69 Sep 11 18:09:14.302308 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.302278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx"] Sep 11 18:09:14.313607 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:09:14.313580 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde58723_a523_4089_a206_d127438341e7.slice/crio-3ef2368036d41eea2265bdf1a722afac2ae781073f0b0f2c3b75f20579379d26 WatchSource:0}: Error finding container 3ef2368036d41eea2265bdf1a722afac2ae781073f0b0f2c3b75f20579379d26: Status 404 returned error can't find the container with id 3ef2368036d41eea2265bdf1a722afac2ae781073f0b0f2c3b75f20579379d26 Sep 11 18:09:14.723931 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.723893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" event={"ID":"5aa7e02e-6e6b-4418-84f3-7917ce7d4897","Type":"ContainerStarted","Data":"266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341"} Sep 11 18:09:14.724117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.723956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" event={"ID":"5aa7e02e-6e6b-4418-84f3-7917ce7d4897","Type":"ContainerStarted","Data":"b376a48053189cb9bc437c4e2c9083895ebb85b0376e092263bd7883f0f3fc69"} Sep 11 18:09:14.724117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.724066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:14.725640 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.725614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" event={"ID":"dde58723-a523-4089-a206-d127438341e7","Type":"ContainerStarted","Data":"5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48"} Sep 11 18:09:14.725757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.725666 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:14.725757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.725689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" event={"ID":"dde58723-a523-4089-a206-d127438341e7","Type":"ContainerStarted","Data":"3ef2368036d41eea2265bdf1a722afac2ae781073f0b0f2c3b75f20579379d26"} Sep 11 18:09:14.746410 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.746365 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" podStartSLOduration=1.746346701 podStartE2EDuration="1.746346701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:14.744736432 +0000 UTC m=+898.775797684" watchObservedRunningTime="2026-09-11 18:09:14.746346701 +0000 UTC m=+898.777407931" Sep 11 18:09:14.767384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:14.767331 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" podStartSLOduration=1.767310411 podStartE2EDuration="1.767310411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:14.767050303 +0000 UTC m=+898.798111535" watchObservedRunningTime="2026-09-11 18:09:14.767310411 +0000 UTC m=+898.798371644" Sep 11 18:09:15.313814 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:15.313775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:16.054323 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:16.054289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:25.732658 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:25.732629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:25.733071 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:25.732703 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:29.866100 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:29.866060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:30.802883 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:30.802846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:31.721916 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:31.721878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:32.591705 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:32.591675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:33.667849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:33.667818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:34.369832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:34.369799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:35.241335 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:35.241304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:36.251589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:36.251551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:37.482341 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:37.482306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:38.030121 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:38.030088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:38.970270 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:38.970240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:39.717161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:39.717130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:40.778875 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:40.778840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:41.471153 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.471122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:41.519663 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.519622 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8"] Sep 11 18:09:41.519888 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.519841 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" podUID="d23de337-a2e9-4eed-8701-575f00f87636" containerName="payload-pre-processing" containerID="cri-o://cefa6a40ab0264f9f69891bcee4550d7d02030f152ee2c86c7756fd2fe7de315" gracePeriod=30 Sep 11 18:09:41.616541 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.616509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:41.758597 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.758508 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm"] Sep 11 18:09:41.759313 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.758764 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" podUID="5aa7e02e-6e6b-4418-84f3-7917ce7d4897" containerName="payload-processing" containerID="cri-o://266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341" gracePeriod=30 Sep 11 18:09:41.774209 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.774181 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx"] Sep 11 18:09:41.774525 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.774499 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" podUID="dde58723-a523-4089-a206-d127438341e7" containerName="payload-pre-processing" containerID="cri-o://5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48" gracePeriod=30 Sep 11 18:09:41.792091 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.792059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:41.866931 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.866902 2579 generic.go:356] "Generic (PLEG): container finished" podID="d23de337-a2e9-4eed-8701-575f00f87636" containerID="cefa6a40ab0264f9f69891bcee4550d7d02030f152ee2c86c7756fd2fe7de315" exitCode=0 Sep 11 18:09:41.867091 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.866937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" event={"ID":"d23de337-a2e9-4eed-8701-575f00f87636","Type":"ContainerDied","Data":"cefa6a40ab0264f9f69891bcee4550d7d02030f152ee2c86c7756fd2fe7de315"} Sep 11 18:09:41.900103 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.900065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:41.913027 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:41.913002 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:09:42.013729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.013648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:42.044665 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.044625 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d23de337-a2e9-4eed-8701-575f00f87636-plugins-config-volume\") pod \"d23de337-a2e9-4eed-8701-575f00f87636\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " Sep 11 18:09:42.044850 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.044712 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw4nl\" (UniqueName: \"kubernetes.io/projected/d23de337-a2e9-4eed-8701-575f00f87636-kube-api-access-rw4nl\") pod \"d23de337-a2e9-4eed-8701-575f00f87636\" (UID: \"d23de337-a2e9-4eed-8701-575f00f87636\") " Sep 11 18:09:42.045031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.045006 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23de337-a2e9-4eed-8701-575f00f87636-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d23de337-a2e9-4eed-8701-575f00f87636" (UID: "d23de337-a2e9-4eed-8701-575f00f87636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:09:42.047183 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.047152 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23de337-a2e9-4eed-8701-575f00f87636-kube-api-access-rw4nl" (OuterVolumeSpecName: "kube-api-access-rw4nl") pod "d23de337-a2e9-4eed-8701-575f00f87636" (UID: "d23de337-a2e9-4eed-8701-575f00f87636"). InnerVolumeSpecName "kube-api-access-rw4nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:09:42.065088 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.065063 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:42.068269 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.068247 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:42.145706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.145671 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5cv9\" (UniqueName: \"kubernetes.io/projected/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-kube-api-access-v5cv9\") pod \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " Sep 11 18:09:42.145849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.145790 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-plugins-config-volume\") pod \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\" (UID: \"5aa7e02e-6e6b-4418-84f3-7917ce7d4897\") " Sep 11 18:09:42.145849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.145832 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d57gq\" (UniqueName: \"kubernetes.io/projected/dde58723-a523-4089-a206-d127438341e7-kube-api-access-d57gq\") pod \"dde58723-a523-4089-a206-d127438341e7\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " Sep 11 18:09:42.145959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.145874 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde58723-a523-4089-a206-d127438341e7-plugins-config-volume\") pod \"dde58723-a523-4089-a206-d127438341e7\" (UID: \"dde58723-a523-4089-a206-d127438341e7\") " Sep 11 18:09:42.146452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.146087 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d23de337-a2e9-4eed-8701-575f00f87636-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:42.146452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.146108 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4nl\" (UniqueName: \"kubernetes.io/projected/d23de337-a2e9-4eed-8701-575f00f87636-kube-api-access-rw4nl\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:42.146452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.146187 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5aa7e02e-6e6b-4418-84f3-7917ce7d4897" (UID: "5aa7e02e-6e6b-4418-84f3-7917ce7d4897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:09:42.146590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.146448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde58723-a523-4089-a206-d127438341e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dde58723-a523-4089-a206-d127438341e7" (UID: "dde58723-a523-4089-a206-d127438341e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:09:42.148091 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.148056 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-kube-api-access-v5cv9" (OuterVolumeSpecName: "kube-api-access-v5cv9") pod "5aa7e02e-6e6b-4418-84f3-7917ce7d4897" (UID: "5aa7e02e-6e6b-4418-84f3-7917ce7d4897"). InnerVolumeSpecName "kube-api-access-v5cv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:09:42.148351 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.148332 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde58723-a523-4089-a206-d127438341e7-kube-api-access-d57gq" (OuterVolumeSpecName: "kube-api-access-d57gq") pod "dde58723-a523-4089-a206-d127438341e7" (UID: "dde58723-a523-4089-a206-d127438341e7"). InnerVolumeSpecName "kube-api-access-d57gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:09:42.187884 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.187853 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:42.247362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.247331 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:42.247362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.247356 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57gq\" (UniqueName: \"kubernetes.io/projected/dde58723-a523-4089-a206-d127438341e7-kube-api-access-d57gq\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:42.247362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.247366 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde58723-a523-4089-a206-d127438341e7-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:42.247621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.247374 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5cv9\" (UniqueName: \"kubernetes.io/projected/5aa7e02e-6e6b-4418-84f3-7917ce7d4897-kube-api-access-v5cv9\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:09:42.383713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.383675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:42.522029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.522002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:42.873103 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.873075 2579 generic.go:356] "Generic (PLEG): container finished" podID="5aa7e02e-6e6b-4418-84f3-7917ce7d4897" containerID="266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341" exitCode=0 Sep 11 18:09:42.873589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.873136 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" Sep 11 18:09:42.873589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.873155 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" event={"ID":"5aa7e02e-6e6b-4418-84f3-7917ce7d4897","Type":"ContainerDied","Data":"266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341"} Sep 11 18:09:42.873589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.873202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm" event={"ID":"5aa7e02e-6e6b-4418-84f3-7917ce7d4897","Type":"ContainerDied","Data":"b376a48053189cb9bc437c4e2c9083895ebb85b0376e092263bd7883f0f3fc69"} Sep 11 18:09:42.873589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.873228 2579 scope.go:117] "RemoveContainer" containerID="266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341" Sep 11 18:09:42.874610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.874490 2579 generic.go:356] "Generic (PLEG): container finished" podID="dde58723-a523-4089-a206-d127438341e7" containerID="5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48" exitCode=0 Sep 11 18:09:42.874610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.874544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" event={"ID":"dde58723-a523-4089-a206-d127438341e7","Type":"ContainerDied","Data":"5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48"} Sep 11 18:09:42.874610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.874571 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" Sep 11 18:09:42.874610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.874593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx" event={"ID":"dde58723-a523-4089-a206-d127438341e7","Type":"ContainerDied","Data":"3ef2368036d41eea2265bdf1a722afac2ae781073f0b0f2c3b75f20579379d26"} Sep 11 18:09:42.876114 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.876093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" event={"ID":"d23de337-a2e9-4eed-8701-575f00f87636","Type":"ContainerDied","Data":"e058515d05bbf8c73b10b75a40c07943bf76ae16cabd0c08b25a579d2e6885c3"} Sep 11 18:09:42.876241 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.876126 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8" Sep 11 18:09:42.884997 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.884964 2579 scope.go:117] "RemoveContainer" containerID="266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341" Sep 11 18:09:42.885280 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:09:42.885262 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341\": container with ID starting with 266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341 not found: ID does not exist" containerID="266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341" Sep 11 18:09:42.885339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.885288 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341"} err="failed to get container status \"266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341\": rpc error: code = NotFound desc = could not find container \"266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341\": container with ID starting with 266d1d7ffa534ccee87f8f1669a61352379e82942a0a0ee7ed8921a6c814f341 not found: ID does not exist" Sep 11 18:09:42.885339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.885304 2579 scope.go:117] "RemoveContainer" containerID="5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48" Sep 11 18:09:42.893295 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.893272 2579 scope.go:117] "RemoveContainer" containerID="5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48" Sep 11 18:09:42.893647 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:09:42.893630 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48\": container with ID starting with 5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48 not found: ID does not exist" containerID="5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48" Sep 11 18:09:42.893713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.893652 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48"} err="failed to get container status \"5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48\": rpc error: code = NotFound desc = could not find container \"5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48\": container with ID starting with 5fcebc8d0d14f61322e9043d8b3c898aee4f49003513e3b77cd9ba03282adf48 not found: ID does not exist" Sep 11 18:09:42.893713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.893673 2579 scope.go:117] "RemoveContainer" containerID="cefa6a40ab0264f9f69891bcee4550d7d02030f152ee2c86c7756fd2fe7de315" Sep 11 18:09:42.905125 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.905105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8"] Sep 11 18:09:42.907031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.907012 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-736ba3-575f5cb8b-s2kb8"] Sep 11 18:09:42.920390 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.920363 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm"] Sep 11 18:09:42.924825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.924804 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-820e15b2-57bddd4cb5-ghpnm"] Sep 11 18:09:42.937699 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.937679 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx"] Sep 11 18:09:42.942931 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:42.942910 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-820e15b2-77b5567ff8-dz8dx"] Sep 11 18:09:43.365662 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:43.365637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:43.521049 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:43.521015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:43.810178 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:43.810095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:44.076059 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.075963 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7"] Sep 11 18:09:44.077189 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077164 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa7e02e-6e6b-4418-84f3-7917ce7d4897" containerName="payload-processing" Sep 11 18:09:44.077189 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077188 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa7e02e-6e6b-4418-84f3-7917ce7d4897" containerName="payload-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077201 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23de337-a2e9-4eed-8701-575f00f87636" containerName="payload-pre-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077209 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23de337-a2e9-4eed-8701-575f00f87636" containerName="payload-pre-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077237 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde58723-a523-4089-a206-d127438341e7" containerName="payload-pre-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077246 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde58723-a523-4089-a206-d127438341e7" containerName="payload-pre-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077340 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde58723-a523-4089-a206-d127438341e7" containerName="payload-pre-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077354 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23de337-a2e9-4eed-8701-575f00f87636" containerName="payload-pre-processing" Sep 11 18:09:44.077389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.077364 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa7e02e-6e6b-4418-84f3-7917ce7d4897" containerName="payload-processing" Sep 11 18:09:44.082082 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.082061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.086706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.086687 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-85c5a226-openshift-default-dockercfg-t9w9f\"" Sep 11 18:09:44.097923 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.097890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7"] Sep 11 18:09:44.109296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.109265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:44.169139 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-podinfo\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4bk\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-kube-api-access-4g4bk\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-certs\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-envoy\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169561 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee475d1e-7a47-453c-bc53-9b92b2809338-istiod-ca-cert\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169561 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-socket\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169561 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-data\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169719 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-credential-socket\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.169719 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.169635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-token\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-podinfo\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4bk\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-kube-api-access-4g4bk\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-certs\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-envoy\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee475d1e-7a47-453c-bc53-9b92b2809338-istiod-ca-cert\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-socket\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.270966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-data\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.270979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-credential-socket\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.271020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-token\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.271251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-certs\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271538 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.271516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-socket\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271604 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.271566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-credential-socket\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271824 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.271624 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-data\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.271824 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.271640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee475d1e-7a47-453c-bc53-9b92b2809338-istiod-ca-cert\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.273302 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.273282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-envoy\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.273709 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.273693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-podinfo\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.281302 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.281277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4bk\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-kube-api-access-4g4bk\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.283546 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.283524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-token\") pod \"e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.284702 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.284680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:44.393458 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.393403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:44.445695 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.445652 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa7e02e-6e6b-4418-84f3-7917ce7d4897" path="/var/lib/kubelet/pods/5aa7e02e-6e6b-4418-84f3-7917ce7d4897/volumes" Sep 11 18:09:44.446717 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.446258 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23de337-a2e9-4eed-8701-575f00f87636" path="/var/lib/kubelet/pods/d23de337-a2e9-4eed-8701-575f00f87636/volumes" Sep 11 18:09:44.446837 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.446802 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde58723-a523-4089-a206-d127438341e7" path="/var/lib/kubelet/pods/dde58723-a523-4089-a206-d127438341e7/volumes" Sep 11 18:09:44.486937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.486900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:44.679640 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.679615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7"] Sep 11 18:09:44.681267 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:09:44.681243 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee475d1e_7a47_453c_bc53_9b92b2809338.slice/crio-e3c3fcc5fdcf2c53d096df768c670be9ccda1d82ea146fe1e4656d0b4a2565db WatchSource:0}: Error finding container e3c3fcc5fdcf2c53d096df768c670be9ccda1d82ea146fe1e4656d0b4a2565db: Status 404 returned error can't find the container with id e3c3fcc5fdcf2c53d096df768c670be9ccda1d82ea146fe1e4656d0b4a2565db Sep 11 18:09:44.683411 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.683383 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:09:44.683511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.683463 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:09:44.683511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.683494 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:09:44.875567 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.875535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:44.891949 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.891914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" event={"ID":"ee475d1e-7a47-453c-bc53-9b92b2809338","Type":"ContainerStarted","Data":"973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef"} Sep 11 18:09:44.892136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.891977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" event={"ID":"ee475d1e-7a47-453c-bc53-9b92b2809338","Type":"ContainerStarted","Data":"e3c3fcc5fdcf2c53d096df768c670be9ccda1d82ea146fe1e4656d0b4a2565db"} Sep 11 18:09:44.917941 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:44.917865 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podStartSLOduration=0.917840356 podStartE2EDuration="917.840356ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:44.915148977 +0000 UTC m=+928.946210208" watchObservedRunningTime="2026-09-11 18:09:44.917840356 +0000 UTC m=+928.948901586" Sep 11 18:09:45.393613 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:45.393575 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:45.398823 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:45.398797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:45.897451 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:45.897387 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:45.904738 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:45.904710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:09:45.983768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:45.983733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:46.424675 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:46.424642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:46.547157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:46.547125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:46.918630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:46.918597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:47.301516 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:47.301402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:48.095975 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.095944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:48.226681 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.226652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:48.609156 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.609114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:48.738405 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.738369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:48.804281 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.804245 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t"] Sep 11 18:09:48.807936 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.807918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:48.810835 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.810815 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-85c5a226\"" Sep 11 18:09:48.810835 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.810832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-85c5a226-dockercfg-ct78r\"" Sep 11 18:09:48.817883 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.817859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t"] Sep 11 18:09:48.865321 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.865239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:48.916897 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.916863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:48.917086 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:48.917034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fnc\" (UniqueName: \"kubernetes.io/projected/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-kube-api-access-s6fnc\") pod \"payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.018035 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.017997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fnc\" (UniqueName: \"kubernetes.io/projected/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-kube-api-access-s6fnc\") pod \"payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.018199 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.018091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.018855 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.018831 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.031440 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.031395 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fnc\" (UniqueName: \"kubernetes.io/projected/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-kube-api-access-s6fnc\") pod \"payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.112728 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.112690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:49.121147 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.121125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.406464 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.406434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t"] Sep 11 18:09:49.408307 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:09:49.408283 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7dbe18_f347_45fd_9a85_67d3d538c6c9.slice/crio-1afc8a2885c28ad65d54553b769a81ce56a1add1dabaa537859ae03fddd0ad82 WatchSource:0}: Error finding container 1afc8a2885c28ad65d54553b769a81ce56a1add1dabaa537859ae03fddd0ad82: Status 404 returned error can't find the container with id 1afc8a2885c28ad65d54553b769a81ce56a1add1dabaa537859ae03fddd0ad82 Sep 11 18:09:49.920571 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.920531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" event={"ID":"9f7dbe18-f347-45fd-9a85-67d3d538c6c9","Type":"ContainerStarted","Data":"c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f"} Sep 11 18:09:49.920768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.920586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" event={"ID":"9f7dbe18-f347-45fd-9a85-67d3d538c6c9","Type":"ContainerStarted","Data":"1afc8a2885c28ad65d54553b769a81ce56a1add1dabaa537859ae03fddd0ad82"} Sep 11 18:09:49.920768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.920630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:09:49.944589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:49.944537 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" podStartSLOduration=1.9445206019999999 podStartE2EDuration="1.944520602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:49.942560183 +0000 UTC m=+933.973621429" watchObservedRunningTime="2026-09-11 18:09:49.944520602 +0000 UTC m=+933.975581831" Sep 11 18:09:50.248729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:50.248636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:51.056190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:51.056097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:51.821476 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:51.821399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:09:55.626123 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:09:55.626086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:00.931952 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:00.931920 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:10:10.052763 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.052729 2579 generic.go:356] "Generic (PLEG): container finished" podID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerID="973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef" exitCode=0 Sep 11 18:10:10.053220 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.052807 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" event={"ID":"ee475d1e-7a47-453c-bc53-9b92b2809338","Type":"ContainerDied","Data":"973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef"} Sep 11 18:10:10.053269 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.053237 2579 scope.go:117] "RemoveContainer" containerID="973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef" Sep 11 18:10:10.054444 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.054393 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:10.054523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.054476 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:10.054523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.054506 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:10.385490 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:10.385452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:11.063101 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:11.062112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" event={"ID":"ee475d1e-7a47-453c-bc53-9b92b2809338","Type":"ContainerStarted","Data":"d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59"} Sep 11 18:10:11.386519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:11.386483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:11.394380 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:11.394352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:10:11.395377 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:11.395357 2579 patch_prober.go:28] interesting pod/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 11 18:10:11.395454 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:11.395434 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 11 18:10:12.262251 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:12.262216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:12.394632 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:12.394596 2579 patch_prober.go:28] interesting pod/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 11 18:10:12.394802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:12.394670 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 11 18:10:13.035622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:13.035585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:13.394615 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:13.394578 2579 patch_prober.go:28] interesting pod/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 11 18:10:13.395035 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:13.394631 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 11 18:10:13.861689 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:13.861555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:14.394267 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:14.394223 2579 patch_prober.go:28] interesting pod/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 11 18:10:14.394510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:14.394286 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 11 18:10:14.775504 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:14.773950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:15.394707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:15.394671 2579 patch_prober.go:28] interesting pod/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 11 18:10:15.394890 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:15.394734 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 11 18:10:15.733402 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:15.733288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:15.898076 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:15.898044 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:10:16.398867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:16.398838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:10:16.399900 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:16.399878 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:10:16.738285 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:16.738208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:18.021190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.021160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:18.050698 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.050668 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t"] Sep 11 18:10:18.050931 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.050883 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" podUID="9f7dbe18-f347-45fd-9a85-67d3d538c6c9" containerName="payload-processing" containerID="cri-o://c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f" gracePeriod=30 Sep 11 18:10:18.162335 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.162295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:18.332158 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.332135 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:10:18.406929 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.406872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:18.434691 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.434651 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-plugins-config-volume\") pod \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " Sep 11 18:10:18.434691 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.434690 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6fnc\" (UniqueName: \"kubernetes.io/projected/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-kube-api-access-s6fnc\") pod \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\" (UID: \"9f7dbe18-f347-45fd-9a85-67d3d538c6c9\") " Sep 11 18:10:18.435108 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.435079 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f7dbe18-f347-45fd-9a85-67d3d538c6c9" (UID: "9f7dbe18-f347-45fd-9a85-67d3d538c6c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:10:18.437096 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.437062 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-kube-api-access-s6fnc" (OuterVolumeSpecName: "kube-api-access-s6fnc") pod "9f7dbe18-f347-45fd-9a85-67d3d538c6c9" (UID: "9f7dbe18-f347-45fd-9a85-67d3d538c6c9"). InnerVolumeSpecName "kube-api-access-s6fnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:10:18.518135 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.518092 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7"] Sep 11 18:10:18.518367 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.518343 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" containerID="cri-o://d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59" gracePeriod=30 Sep 11 18:10:18.535673 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.535637 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:18.535673 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.535672 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fnc\" (UniqueName: \"kubernetes.io/projected/9f7dbe18-f347-45fd-9a85-67d3d538c6c9-kube-api-access-s6fnc\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:18.645143 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.645109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:18.999886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:18.999803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:19.115548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.115512 2579 generic.go:356] "Generic (PLEG): container finished" podID="9f7dbe18-f347-45fd-9a85-67d3d538c6c9" containerID="c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f" exitCode=0 Sep 11 18:10:19.115943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.115563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" event={"ID":"9f7dbe18-f347-45fd-9a85-67d3d538c6c9","Type":"ContainerDied","Data":"c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f"} Sep 11 18:10:19.115943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.115650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" event={"ID":"9f7dbe18-f347-45fd-9a85-67d3d538c6c9","Type":"ContainerDied","Data":"1afc8a2885c28ad65d54553b769a81ce56a1add1dabaa537859ae03fddd0ad82"} Sep 11 18:10:19.115943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.115581 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t" Sep 11 18:10:19.115943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.115680 2579 scope.go:117] "RemoveContainer" containerID="c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f" Sep 11 18:10:19.124881 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.124856 2579 scope.go:117] "RemoveContainer" containerID="c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f" Sep 11 18:10:19.125210 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:10:19.125186 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f\": container with ID starting with c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f not found: ID does not exist" containerID="c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f" Sep 11 18:10:19.125309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.125222 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f"} err="failed to get container status \"c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f\": rpc error: code = NotFound desc = could not find container \"c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f\": container with ID starting with c493c67f15fd432ed564a8cd148c16eba6d6e25aaaba678f3c5168c4ccfe695f not found: ID does not exist" Sep 11 18:10:19.160527 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.160493 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t"] Sep 11 18:10:19.188994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.188964 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-85c5a226-5b6d8699fc-zwk4t"] Sep 11 18:10:19.300259 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.300171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:19.474373 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:19.474338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:20.182265 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:20.182233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:20.444718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:20.444615 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7dbe18-f347-45fd-9a85-67d3d538c6c9" path="/var/lib/kubelet/pods/9f7dbe18-f347-45fd-9a85-67d3d538c6c9/volumes" Sep 11 18:10:20.978357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:20.978324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:21.273594 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:21.273475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:21.940038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:21.939986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:22.920371 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:22.920336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:23.778035 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.778014 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:10:23.887622 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887588 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-certs\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887693 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-podinfo\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887729 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-token\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-socket\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887832 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g4bk\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-kube-api-access-4g4bk\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-envoy\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887865 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:10:23.887978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-credential-socket\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.887978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887941 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-data\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.888219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.887982 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee475d1e-7a47-453c-bc53-9b92b2809338-istiod-ca-cert\") pod \"ee475d1e-7a47-453c-bc53-9b92b2809338\" (UID: \"ee475d1e-7a47-453c-bc53-9b92b2809338\") " Sep 11 18:10:23.888219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888096 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:10:23.888219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:10:23.888379 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888329 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:10:23.888379 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888346 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.888379 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888365 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.888575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888380 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.888575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.888483 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee475d1e-7a47-453c-bc53-9b92b2809338-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:10:23.890279 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.890248 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:10:23.890403 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.890349 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:10:23.890571 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.890549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:10:23.890617 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.890553 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-kube-api-access-4g4bk" (OuterVolumeSpecName: "kube-api-access-4g4bk") pod "ee475d1e-7a47-453c-bc53-9b92b2809338" (UID: "ee475d1e-7a47-453c-bc53-9b92b2809338"). InnerVolumeSpecName "kube-api-access-4g4bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:10:23.989926 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.989896 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.989926 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.989923 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee475d1e-7a47-453c-bc53-9b92b2809338-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.989926 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.989932 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.990352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.989939 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.990352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.989948 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4bk\" (UniqueName: \"kubernetes.io/projected/ee475d1e-7a47-453c-bc53-9b92b2809338-kube-api-access-4g4bk\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:23.990352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:23.989958 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee475d1e-7a47-453c-bc53-9b92b2809338-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:10:24.144196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.144101 2579 generic.go:356] "Generic (PLEG): container finished" podID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerID="d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59" exitCode=0 Sep 11 18:10:24.144196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.144157 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" Sep 11 18:10:24.144196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.144187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" event={"ID":"ee475d1e-7a47-453c-bc53-9b92b2809338","Type":"ContainerDied","Data":"d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59"} Sep 11 18:10:24.144510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.144240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7" event={"ID":"ee475d1e-7a47-453c-bc53-9b92b2809338","Type":"ContainerDied","Data":"e3c3fcc5fdcf2c53d096df768c670be9ccda1d82ea146fe1e4656d0b4a2565db"} Sep 11 18:10:24.144510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.144263 2579 scope.go:117] "RemoveContainer" containerID="d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59" Sep 11 18:10:24.153523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.153508 2579 scope.go:117] "RemoveContainer" containerID="973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef" Sep 11 18:10:24.161326 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.161309 2579 scope.go:117] "RemoveContainer" containerID="d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59" Sep 11 18:10:24.161617 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:10:24.161600 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59\": container with ID starting with d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59 not found: ID does not exist" containerID="d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59" Sep 11 18:10:24.161687 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.161626 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59"} err="failed to get container status \"d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59\": rpc error: code = NotFound desc = could not find container \"d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59\": container with ID starting with d2b9fe20c7d0adc23f1df45186e3c903ac57850b30c1921aa397224fcb0fbf59 not found: ID does not exist" Sep 11 18:10:24.161687 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.161644 2579 scope.go:117] "RemoveContainer" containerID="973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef" Sep 11 18:10:24.161868 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:10:24.161850 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef\": container with ID starting with 973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef not found: ID does not exist" containerID="973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef" Sep 11 18:10:24.161912 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.161874 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef"} err="failed to get container status \"973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef\": rpc error: code = NotFound desc = could not find container \"973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef\": container with ID starting with 973837dae1b0ecd7105f1ce89c1d39508a65a30b7c11ebfdf081dfb1e18f05ef not found: ID does not exist" Sep 11 18:10:24.168808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.168785 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7"] Sep 11 18:10:24.173747 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.173727 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-85c5a226-openshift-default-54df4d46b8-nrmj7"] Sep 11 18:10:24.444875 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.444794 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" path="/var/lib/kubelet/pods/ee475d1e-7a47-453c-bc53-9b92b2809338/volumes" Sep 11 18:10:24.774738 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:24.774663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:26.534484 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:26.534453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:28.011800 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:28.011750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:28.123087 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:28.123051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:28.237725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:28.237693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:28.344378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:28.344298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:28.704366 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:28.704335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:33.625281 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:33.625252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:33.732714 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:33.732685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:33.930896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:33.930763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:34.393693 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.393660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:34.724075 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.723994 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq"] Sep 11 18:10:34.725100 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.725074 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7dbe18-f347-45fd-9a85-67d3d538c6c9" containerName="payload-processing" Sep 11 18:10:34.725100 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.725102 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7dbe18-f347-45fd-9a85-67d3d538c6c9" containerName="payload-processing" Sep 11 18:10:34.725261 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.725133 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" Sep 11 18:10:34.725261 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.725142 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" Sep 11 18:10:34.725261 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.725241 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7dbe18-f347-45fd-9a85-67d3d538c6c9" containerName="payload-processing" Sep 11 18:10:34.725261 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.725254 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" Sep 11 18:10:34.728790 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.728769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.732839 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.732806 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-aedf1a-openshift-default-dockercfg-qp5vw\"" Sep 11 18:10:34.743772 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.743750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq"] Sep 11 18:10:34.748923 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.748890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:34.893790 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.893753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-envoy\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.893965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.893894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istiod-ca-cert\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.893965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.893938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rzl\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-kube-api-access-59rzl\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.894059 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.893966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-socket\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.894059 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.893988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-data\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.894059 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.894007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-podinfo\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.894059 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.894041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-credential-socket\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.894191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.894121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-certs\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.894191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.894165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-token\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istiod-ca-cert\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rzl\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-kube-api-access-59rzl\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-socket\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-data\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-podinfo\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-credential-socket\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-certs\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-token\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.995819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-envoy\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.996126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-data\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.996126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.995970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-socket\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.996126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.996028 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-credential-socket\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.996262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.996144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-certs\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.996262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.996204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istiod-ca-cert\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.998165 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.998143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-envoy\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:34.998387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:34.998367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-podinfo\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:35.007998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.007975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-token\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:35.009404 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.009360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rzl\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-kube-api-access-59rzl\") pod \"e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:35.040328 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.040299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:35.081520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.081490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:35.326742 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.326714 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq"] Sep 11 18:10:35.327598 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:10:35.327574 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b9ff1c_e71d_4251_99c5_6778e4f4f2b5.slice/crio-0458fe5c7882b9fc0b0ee39df5f46b79431243968bd0924c5cc8480403454ee5 WatchSource:0}: Error finding container 0458fe5c7882b9fc0b0ee39df5f46b79431243968bd0924c5cc8480403454ee5: Status 404 returned error can't find the container with id 0458fe5c7882b9fc0b0ee39df5f46b79431243968bd0924c5cc8480403454ee5 Sep 11 18:10:35.329770 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.329739 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:35.329831 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.329808 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:35.329865 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:35.329856 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:36.207459 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:36.207390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" event={"ID":"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5","Type":"ContainerStarted","Data":"200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099"} Sep 11 18:10:36.207959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:36.207475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" event={"ID":"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5","Type":"ContainerStarted","Data":"0458fe5c7882b9fc0b0ee39df5f46b79431243968bd0924c5cc8480403454ee5"} Sep 11 18:10:36.232965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:36.232921 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podStartSLOduration=2.232907357 podStartE2EDuration="2.232907357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:36.230097886 +0000 UTC m=+980.261159127" watchObservedRunningTime="2026-09-11 18:10:36.232907357 +0000 UTC m=+980.263968587" Sep 11 18:10:37.040645 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:37.040611 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:37.046349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:37.046328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:37.213177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:37.213142 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:37.214233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:37.214210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:10:37.838296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:37.838210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:38.231555 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:38.231516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:38.327454 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:38.327399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:38.481312 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:38.481281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:39.452155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:39.452116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:40.332697 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:40.332647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:41.118941 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:41.118906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:42.061562 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:42.061530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:42.159779 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:42.159722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:42.598530 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:42.598493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:42.794522 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:42.794479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:50.333694 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:50.333663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:54.687042 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:54.687007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:10:59.334916 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:59.334881 2579 generic.go:356] "Generic (PLEG): container finished" podID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerID="200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099" exitCode=0 Sep 11 18:10:59.334916 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:59.334918 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" event={"ID":"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5","Type":"ContainerDied","Data":"200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099"} Sep 11 18:10:59.335438 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:59.335315 2579 scope.go:117] "RemoveContainer" containerID="200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099" Sep 11 18:10:59.336494 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:59.336462 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:59.336547 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:59.336533 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:10:59.336599 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:10:59.336586 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:11:00.341980 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:00.341947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" event={"ID":"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5","Type":"ContainerStarted","Data":"e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47"} Sep 11 18:11:01.041068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:01.041023 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:11:01.041298 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:01.041271 2579 patch_prober.go:28] interesting pod/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 11 18:11:01.041383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:01.041334 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 11 18:11:02.041197 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:02.041159 2579 patch_prober.go:28] interesting pod/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 11 18:11:02.041689 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:02.041222 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 11 18:11:02.296098 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:02.295997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:02.437475 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:02.437434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:02.537808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:02.537777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:03.040756 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:03.040713 2579 patch_prober.go:28] interesting pod/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 11 18:11:03.040942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:03.040779 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 11 18:11:04.040681 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:04.040651 2579 patch_prober.go:28] interesting pod/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 11 18:11:04.041123 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:04.041098 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 11 18:11:04.894020 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:04.893976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:05.041384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:05.041348 2579 patch_prober.go:28] interesting pod/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 11 18:11:05.041781 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:05.041400 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 11 18:11:05.883413 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:05.883372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:06.045344 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:06.045313 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:11:06.045771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:06.045697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:11:06.046466 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:06.046450 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:11:06.758013 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:06.757966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:07.460557 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:07.460526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:08.248703 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:08.248664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:08.354838 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:08.354803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:09.110483 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:09.110449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:18.335253 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:18.335223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:24.810219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:24.810188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:24.911473 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:24.911413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:25.013585 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:25.013547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:29.622596 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:29.622564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:35.524801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:35.524737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:36.322720 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:36.322669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:43.081041 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:43.081000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:43.182643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:43.182611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:43.281984 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:43.281952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:43.385592 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:43.385563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:44.001949 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:44.001909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:44.118413 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:44.118383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:44.203604 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:44.203572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:45.738849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:45.738817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:52.046520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:52.046488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:52.166214 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:52.166185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:52.714671 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:52.714633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:59.408894 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:59.408853 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:59.510211 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:59.510179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:59.636764 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:59.636732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:11:59.748029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:11:59.747948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:00.026780 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.026681 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth"] Sep 11 18:12:00.027120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.027107 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" Sep 11 18:12:00.027185 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.027121 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" Sep 11 18:12:00.027241 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.027188 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee475d1e-7a47-453c-bc53-9b92b2809338" containerName="istio-proxy" Sep 11 18:12:00.029518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.029501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.032765 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.032742 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7c2684c9-openshift-default-dockercfg-ctw6r\"" Sep 11 18:12:00.036828 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.036804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:00.051682 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.051648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth"] Sep 11 18:12:00.151380 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-token\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151609 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2z2j\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-kube-api-access-z2z2j\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151609 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-data\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151609 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151561 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-podinfo\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151609 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324ecca3-ed16-4ddc-95b8-ca581107d32e-istiod-ca-cert\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151609 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-credential-socket\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151783 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-envoy\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151783 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-socket\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.151783 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.151731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-certs\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.163927 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.163896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:00.252763 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.252724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-envoy\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.252763 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.252767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-socket\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.252797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-certs\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.252833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-token\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.252928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2z2j\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-kube-api-access-z2z2j\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.252975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-data\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-podinfo\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253282 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324ecca3-ed16-4ddc-95b8-ca581107d32e-istiod-ca-cert\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253282 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-credential-socket\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253282 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253216 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-socket\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253487 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-certs\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253487 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-data\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253487 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-credential-socket\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.253784 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.253767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324ecca3-ed16-4ddc-95b8-ca581107d32e-istiod-ca-cert\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.255181 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.255161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-envoy\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.255814 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.255794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-podinfo\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.263978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.263954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-token\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.264251 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.264228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2z2j\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-kube-api-access-z2z2j\") pod \"e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.341510 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.341402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:00.624880 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.624856 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth"] Sep 11 18:12:00.627874 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:12:00.627849 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324ecca3_ed16_4ddc_95b8_ca581107d32e.slice/crio-27b98941aebc7f771f6a4d2774b39d168302ad15224c2a571ea1555105e44070 WatchSource:0}: Error finding container 27b98941aebc7f771f6a4d2774b39d168302ad15224c2a571ea1555105e44070: Status 404 returned error can't find the container with id 27b98941aebc7f771f6a4d2774b39d168302ad15224c2a571ea1555105e44070 Sep 11 18:12:00.631272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.631063 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:12:00.631272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.631160 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:12:00.631272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.631207 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:12:00.645628 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:00.645595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" event={"ID":"324ecca3-ed16-4ddc-95b8-ca581107d32e","Type":"ContainerStarted","Data":"27b98941aebc7f771f6a4d2774b39d168302ad15224c2a571ea1555105e44070"} Sep 11 18:12:01.652478 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:01.652434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" event={"ID":"324ecca3-ed16-4ddc-95b8-ca581107d32e","Type":"ContainerStarted","Data":"4ead028d18afd350d7ec1cb412660394e23abf1ce2836add6192cd3f8ff8671a"} Sep 11 18:12:01.678792 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:01.678737 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podStartSLOduration=2.678714568 podStartE2EDuration="2.678714568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:01.676898871 +0000 UTC m=+1065.707960115" watchObservedRunningTime="2026-09-11 18:12:01.678714568 +0000 UTC m=+1065.709775799" Sep 11 18:12:02.342495 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:02.342452 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:02.348183 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:02.348151 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:02.658201 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:02.658172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:02.659361 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:02.659340 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:05.636075 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:05.636037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:06.301090 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:06.301040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:06.404227 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:06.404193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:06.778604 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:06.778572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:07.505703 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:07.505667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:08.395885 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:08.395741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:09.154620 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:09.154589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:09.852900 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:09.852862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:10.632703 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:10.632652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:10.756377 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:10.756348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:10.974379 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:10.974297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:11.121948 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.121910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:11.216521 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.216488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:11.284354 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.284278 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx"] Sep 11 18:12:11.286881 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.286863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.290099 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.290079 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7c2684c9\"" Sep 11 18:12:11.290224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.290104 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7c2684c9-dockercfg-tvm2v\"" Sep 11 18:12:11.299157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.299139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx"] Sep 11 18:12:11.363691 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.363658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vs5l\" (UniqueName: \"kubernetes.io/projected/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-kube-api-access-5vs5l\") pod \"payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.363883 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.363745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.465164 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.465124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vs5l\" (UniqueName: \"kubernetes.io/projected/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-kube-api-access-5vs5l\") pod \"payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.465352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.465210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.465994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.465970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.474962 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.474930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vs5l\" (UniqueName: \"kubernetes.io/projected/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-kube-api-access-5vs5l\") pod \"payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.598549 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.598454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:11.874335 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:11.874307 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx"] Sep 11 18:12:11.875852 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:12:11.875830 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0eb8bf2_718f_42e1_9525_0c0e8480ab75.slice/crio-5f5292374bc4d21c071a27380c95040123d00807147cc81f644777d6ebda03c5 WatchSource:0}: Error finding container 5f5292374bc4d21c071a27380c95040123d00807147cc81f644777d6ebda03c5: Status 404 returned error can't find the container with id 5f5292374bc4d21c071a27380c95040123d00807147cc81f644777d6ebda03c5 Sep 11 18:12:12.716911 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:12.716873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" event={"ID":"d0eb8bf2-718f-42e1-9525-0c0e8480ab75","Type":"ContainerStarted","Data":"9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223"} Sep 11 18:12:12.717345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:12.716934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" event={"ID":"d0eb8bf2-718f-42e1-9525-0c0e8480ab75","Type":"ContainerStarted","Data":"5f5292374bc4d21c071a27380c95040123d00807147cc81f644777d6ebda03c5"} Sep 11 18:12:12.717345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:12.716973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:12.736015 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:12.735966 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" podStartSLOduration=1.735951306 podStartE2EDuration="1.735951306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:12.73475687 +0000 UTC m=+1076.765818114" watchObservedRunningTime="2026-09-11 18:12:12.735951306 +0000 UTC m=+1076.767012535" Sep 11 18:12:19.431934 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:19.431900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:21.151698 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:21.151665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:21.253661 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:21.253620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:22.611398 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:22.611360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:22.713688 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:22.713656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:23.723979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:23.723945 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:29.835462 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:29.835431 2579 generic.go:356] "Generic (PLEG): container finished" podID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerID="4ead028d18afd350d7ec1cb412660394e23abf1ce2836add6192cd3f8ff8671a" exitCode=0 Sep 11 18:12:29.835860 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:29.835479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" event={"ID":"324ecca3-ed16-4ddc-95b8-ca581107d32e","Type":"ContainerDied","Data":"4ead028d18afd350d7ec1cb412660394e23abf1ce2836add6192cd3f8ff8671a"} Sep 11 18:12:29.835943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:29.835926 2579 scope.go:117] "RemoveContainer" containerID="4ead028d18afd350d7ec1cb412660394e23abf1ce2836add6192cd3f8ff8671a" Sep 11 18:12:29.837061 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:29.837027 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:12:29.837169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:29.837111 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:12:29.837230 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:29.837181 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:12:30.842324 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:30.842293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" event={"ID":"324ecca3-ed16-4ddc-95b8-ca581107d32e","Type":"ContainerStarted","Data":"749df2715f1e6040e39b87e8a92081695bc4db95492d38c7486fd830e1fc7cf3"} Sep 11 18:12:31.342253 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:31.342217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:31.343568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:31.343544 2579 patch_prober.go:28] interesting pod/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 11 18:12:31.343686 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:31.343592 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 11 18:12:32.342362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:32.342327 2579 patch_prober.go:28] interesting pod/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 11 18:12:32.342775 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:32.342382 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 11 18:12:32.658219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:32.658187 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:33.222643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:33.222606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:33.342707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:33.342671 2579 patch_prober.go:28] interesting pod/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 11 18:12:33.343084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:33.342728 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 11 18:12:34.112497 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:34.112465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:34.341993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:34.341959 2579 patch_prober.go:28] interesting pod/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 11 18:12:34.342161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:34.342015 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 11 18:12:35.109769 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:35.109738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:35.342335 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:35.342304 2579 patch_prober.go:28] interesting pod/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 11 18:12:35.342533 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:35.342359 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 11 18:12:36.080331 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:36.080289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:36.346175 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:36.346104 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:36.347138 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:36.347119 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:12:36.389839 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:36.389807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:37.078730 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:37.078690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:37.165824 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:37.165778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:37.523337 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:37.523293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:37.921174 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:37.921139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:38.028348 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:38.028308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:38.872693 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:38.872660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:39.953677 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:39.953649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:43.258200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:43.258166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:44.426523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:44.426485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:44.527212 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:44.527182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:44.948653 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:44.948618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:45.831279 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:45.831249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:46.669649 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:46.669605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:47.596703 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:47.596664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:48.886408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:48.886374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:50.376083 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:50.376054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:51.469309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:51.469276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:52.232262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:52.232222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:52.386501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:52.386457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:53.081126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:53.081092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:54.977117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:54.977087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:55.543788 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.543750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:55.586689 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.586536 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx"] Sep 11 18:12:55.586914 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.586846 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" podUID="d0eb8bf2-718f-42e1-9525-0c0e8480ab75" containerName="payload-pre-processing" containerID="cri-o://9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223" gracePeriod=30 Sep 11 18:12:55.616444 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.616376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:55.719167 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.719134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:55.777738 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.777657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:55.850024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.850003 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:55.939963 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.939924 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-plugins-config-volume\") pod \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " Sep 11 18:12:55.940570 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.940291 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0eb8bf2-718f-42e1-9525-0c0e8480ab75" (UID: "d0eb8bf2-718f-42e1-9525-0c0e8480ab75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:12:55.940737 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.940224 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vs5l\" (UniqueName: \"kubernetes.io/projected/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-kube-api-access-5vs5l\") pod \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\" (UID: \"d0eb8bf2-718f-42e1-9525-0c0e8480ab75\") " Sep 11 18:12:55.942137 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.942114 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:12:55.947656 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.947608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-kube-api-access-5vs5l" (OuterVolumeSpecName: "kube-api-access-5vs5l") pod "d0eb8bf2-718f-42e1-9525-0c0e8480ab75" (UID: "d0eb8bf2-718f-42e1-9525-0c0e8480ab75"). InnerVolumeSpecName "kube-api-access-5vs5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:12:55.985488 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.985447 2579 generic.go:356] "Generic (PLEG): container finished" podID="d0eb8bf2-718f-42e1-9525-0c0e8480ab75" containerID="9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223" exitCode=0 Sep 11 18:12:55.985488 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.985492 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" Sep 11 18:12:55.986009 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.985455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" event={"ID":"d0eb8bf2-718f-42e1-9525-0c0e8480ab75","Type":"ContainerDied","Data":"9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223"} Sep 11 18:12:55.986009 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.985630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx" event={"ID":"d0eb8bf2-718f-42e1-9525-0c0e8480ab75","Type":"ContainerDied","Data":"5f5292374bc4d21c071a27380c95040123d00807147cc81f644777d6ebda03c5"} Sep 11 18:12:55.986009 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.985673 2579 scope.go:117] "RemoveContainer" containerID="9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223" Sep 11 18:12:55.995177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.995159 2579 scope.go:117] "RemoveContainer" containerID="9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223" Sep 11 18:12:55.995494 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:12:55.995470 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223\": container with ID starting with 9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223 not found: ID does not exist" containerID="9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223" Sep 11 18:12:55.995577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:55.995500 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223"} err="failed to get container status \"9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223\": rpc error: code = NotFound desc = could not find container \"9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223\": container with ID starting with 9059c5665272cd90508f9bdf9f71e24fd1450842a813a51e52215b6f62922223 not found: ID does not exist" Sep 11 18:12:56.010051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:56.010027 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx"] Sep 11 18:12:56.016224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:56.016160 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7c2684c9-788d8b75d6-fncbx"] Sep 11 18:12:56.043368 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:56.043339 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vs5l\" (UniqueName: \"kubernetes.io/projected/d0eb8bf2-718f-42e1-9525-0c0e8480ab75-kube-api-access-5vs5l\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:12:56.104011 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:56.103958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:56.204408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:56.204366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:56.446086 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:56.446046 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0eb8bf2-718f-42e1-9525-0c0e8480ab75" path="/var/lib/kubelet/pods/d0eb8bf2-718f-42e1-9525-0c0e8480ab75/volumes" Sep 11 18:12:57.196898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:57.196858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:58.131155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:58.131128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:59.174684 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:59.174655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:12:59.973565 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:12:59.973510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:00.749274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:00.749242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:02.589202 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:02.589168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:02.655002 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:02.654968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:02.842514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:02.842411 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth"] Sep 11 18:13:02.842683 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:02.842658 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" containerID="cri-o://749df2715f1e6040e39b87e8a92081695bc4db95492d38c7486fd830e1fc7cf3" gracePeriod=30 Sep 11 18:13:02.851617 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:02.851591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:03.200156 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:03.200116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:03.293822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:03.293783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:03.431176 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:03.431143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:06.348713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:06.348682 2579 patch_prober.go:28] interesting pod/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:13:06.349078 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:06.348750 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:13:08.055244 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.055206 2579 generic.go:356] "Generic (PLEG): container finished" podID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerID="749df2715f1e6040e39b87e8a92081695bc4db95492d38c7486fd830e1fc7cf3" exitCode=0 Sep 11 18:13:08.055661 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.055278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" event={"ID":"324ecca3-ed16-4ddc-95b8-ca581107d32e","Type":"ContainerDied","Data":"749df2715f1e6040e39b87e8a92081695bc4db95492d38c7486fd830e1fc7cf3"} Sep 11 18:13:08.055661 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.055353 2579 scope.go:117] "RemoveContainer" containerID="4ead028d18afd350d7ec1cb412660394e23abf1ce2836add6192cd3f8ff8671a" Sep 11 18:13:08.096233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.096204 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:13:08.166975 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.166939 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2z2j\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-kube-api-access-z2z2j\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.166975 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.166972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-data\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167007 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-podinfo\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-socket\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324ecca3-ed16-4ddc-95b8-ca581107d32e-istiod-ca-cert\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-token\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167117 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-credential-socket\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167479 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-certs\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167479 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-envoy\") pod \"324ecca3-ed16-4ddc-95b8-ca581107d32e\" (UID: \"324ecca3-ed16-4ddc-95b8-ca581107d32e\") " Sep 11 18:13:08.167601 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167215 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-data" (OuterVolumeSpecName: "istio-data") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:08.167601 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167486 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:08.167601 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167541 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:08.167601 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324ecca3-ed16-4ddc-95b8-ca581107d32e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:13:08.167807 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167668 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.167807 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167688 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.167807 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167744 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:08.167807 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.167740 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.169476 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.169449 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-kube-api-access-z2z2j" (OuterVolumeSpecName: "kube-api-access-z2z2j") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "kube-api-access-z2z2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:13:08.169615 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.169576 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-token" (OuterVolumeSpecName: "istio-token") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:13:08.169770 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.169747 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:08.169870 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.169805 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "324ecca3-ed16-4ddc-95b8-ca581107d32e" (UID: "324ecca3-ed16-4ddc-95b8-ca581107d32e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:13:08.268593 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.268554 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.268593 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.268585 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2z2j\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-kube-api-access-z2z2j\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.268593 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.268596 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.268593 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.268604 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324ecca3-ed16-4ddc-95b8-ca581107d32e-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.268864 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.268613 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324ecca3-ed16-4ddc-95b8-ca581107d32e-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:08.268864 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:08.268621 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324ecca3-ed16-4ddc-95b8-ca581107d32e-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:09.061526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.061491 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" Sep 11 18:13:09.061526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.061516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth" event={"ID":"324ecca3-ed16-4ddc-95b8-ca581107d32e","Type":"ContainerDied","Data":"27b98941aebc7f771f6a4d2774b39d168302ad15224c2a571ea1555105e44070"} Sep 11 18:13:09.062079 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.061577 2579 scope.go:117] "RemoveContainer" containerID="749df2715f1e6040e39b87e8a92081695bc4db95492d38c7486fd830e1fc7cf3" Sep 11 18:13:09.084829 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.084796 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth"] Sep 11 18:13:09.090401 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.090378 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7c2684c9-openshift-default-757cdf7b84-m2mth"] Sep 11 18:13:09.476068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.476029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:09.534317 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.534270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:09.617629 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.617580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:09.722743 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.722704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:09.819509 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:09.819399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:10.273333 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:10.273301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:10.444429 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:10.444397 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" path="/var/lib/kubelet/pods/324ecca3-ed16-4ddc-95b8-ca581107d32e/volumes" Sep 11 18:13:11.747001 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:11.746972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:12.661383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:12.661335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:12.743176 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:12.743128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:13.384020 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:13.383961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:14.077809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:14.077772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:14.908855 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:14.908818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:15.662938 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:15.662839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:16.574778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:16.574745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:16.685511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:16.685479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:16.785870 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:16.785829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:17.337467 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.337373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:17.552659 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.552627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:17.856679 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.856646 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf"] Sep 11 18:13:17.857099 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857082 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" Sep 11 18:13:17.857146 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857106 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" Sep 11 18:13:17.857146 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857129 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0eb8bf2-718f-42e1-9525-0c0e8480ab75" containerName="payload-pre-processing" Sep 11 18:13:17.857146 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857142 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0eb8bf2-718f-42e1-9525-0c0e8480ab75" containerName="payload-pre-processing" Sep 11 18:13:17.857265 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857254 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" Sep 11 18:13:17.857305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857268 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0eb8bf2-718f-42e1-9525-0c0e8480ab75" containerName="payload-pre-processing" Sep 11 18:13:17.857305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.857276 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" Sep 11 18:13:17.860605 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.860589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.865857 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.865833 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b05610-openshift-default-dockercfg-6shzl\"" Sep 11 18:13:17.887597 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.887572 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf"] Sep 11 18:13:17.906228 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.906199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:17.964908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.964871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkp9d\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-kube-api-access-kkp9d\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.965220 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.965195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-envoy\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.965394 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.965380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-token\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.965590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.965573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istiod-ca-cert\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.965811 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.965795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-credential-socket\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.965977 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.965960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-socket\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.966136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.966119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-certs\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.966294 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.966281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-podinfo\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:17.966449 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:17.966434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-data\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067301 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-socket\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-certs\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-podinfo\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-data\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkp9d\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-kube-api-access-kkp9d\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-envoy\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-token\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istiod-ca-cert\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-credential-socket\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-socket\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.067867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.067854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-credential-socket\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.068128 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.068100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-certs\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.068199 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.068125 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-data\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.068718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.068694 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istiod-ca-cert\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.070013 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.069982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-envoy\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.070209 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.070192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-podinfo\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.078038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.078015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-token\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.078364 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.078338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkp9d\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-kube-api-access-kkp9d\") pod \"e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.150805 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.150772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:18.172003 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.171974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:18.481771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.481746 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf"] Sep 11 18:13:18.482963 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:13:18.482939 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f30f33_f227_43bf_b0cf_bc0b2bb2080f.slice/crio-7aea2aa3ec6f716ba2de6edb3b4c2c998c963f4ac9ccd3e6806d998dc222c47c WatchSource:0}: Error finding container 7aea2aa3ec6f716ba2de6edb3b4c2c998c963f4ac9ccd3e6806d998dc222c47c: Status 404 returned error can't find the container with id 7aea2aa3ec6f716ba2de6edb3b4c2c998c963f4ac9ccd3e6806d998dc222c47c Sep 11 18:13:18.485274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.485244 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:13:18.485372 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.485306 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:13:18.485372 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:18.485334 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:13:19.116021 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:19.115985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" event={"ID":"17f30f33-f227-43bf-b0cf-bc0b2bb2080f","Type":"ContainerStarted","Data":"5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6"} Sep 11 18:13:19.116395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:19.116046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" event={"ID":"17f30f33-f227-43bf-b0cf-bc0b2bb2080f","Type":"ContainerStarted","Data":"7aea2aa3ec6f716ba2de6edb3b4c2c998c963f4ac9ccd3e6806d998dc222c47c"} Sep 11 18:13:19.172509 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:19.172480 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:19.172661 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:19.172489 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podStartSLOduration=2.172472006 podStartE2EDuration="2.172472006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:19.171455628 +0000 UTC m=+1143.202516877" watchObservedRunningTime="2026-09-11 18:13:19.172472006 +0000 UTC m=+1143.203533241" Sep 11 18:13:19.177947 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:19.177925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:20.122446 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:20.122392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:20.123281 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:20.123261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:21.316722 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:21.316687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:21.422061 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:21.422034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:22.105490 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:22.105457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:22.849619 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:22.849588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:23.660062 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:23.660031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:24.485265 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:24.485217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.213447 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.212167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.294688 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.294657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.360761 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.360731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.444607 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.444577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.446354 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.446328 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w"] Sep 11 18:13:25.446916 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.446897 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" Sep 11 18:13:25.447036 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.446919 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="324ecca3-ed16-4ddc-95b8-ca581107d32e" containerName="istio-proxy" Sep 11 18:13:25.449232 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.449213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.452408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.452375 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b05610-serving-cert\"" Sep 11 18:13:25.465319 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.465255 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w"] Sep 11 18:13:25.530399 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.530365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.543693 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.543669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9b2w\" (UniqueName: \"kubernetes.io/projected/c192bb39-b3dc-460c-9ff1-115899dcf9e7-kube-api-access-f9b2w\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.543795 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.543716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.543795 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.543789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.543908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.543886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c192bb39-b3dc-460c-9ff1-115899dcf9e7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.586844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.586813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.645461 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.645407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9b2w\" (UniqueName: \"kubernetes.io/projected/c192bb39-b3dc-460c-9ff1-115899dcf9e7-kube-api-access-f9b2w\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.645680 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.645480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.645680 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.645528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.645680 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:13:25.645639 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-serving-cert: secret "maas-api-e2e-shared-a-w2-b05610-serving-cert" not found Sep 11 18:13:25.645856 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:13:25.645714 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls podName:c192bb39-b3dc-460c-9ff1-115899dcf9e7 nodeName:}" failed. No retries permitted until 2026-09-11 18:13:26.145695596 +0000 UTC m=+1150.176756804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" (UID: "c192bb39-b3dc-460c-9ff1-115899dcf9e7") : secret "maas-api-e2e-shared-a-w2-b05610-serving-cert" not found Sep 11 18:13:25.645856 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.645735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c192bb39-b3dc-460c-9ff1-115899dcf9e7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.646294 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.646272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c192bb39-b3dc-460c-9ff1-115899dcf9e7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.648211 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.648192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.654627 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.654608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9b2w\" (UniqueName: \"kubernetes.io/projected/c192bb39-b3dc-460c-9ff1-115899dcf9e7-kube-api-access-f9b2w\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:25.746957 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.746861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:25.870140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:25.870110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:26.151577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.151540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:26.154222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.154198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:26.187804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.187772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:26.275138 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.275103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:26.363641 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.363612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:26.654406 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.654364 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w"] Sep 11 18:13:26.659701 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.659678 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:13:26.660742 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:26.660718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:27.159908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:27.159873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" event={"ID":"c192bb39-b3dc-460c-9ff1-115899dcf9e7","Type":"ContainerStarted","Data":"af2ccf5ea9cde1ddde4fdb1600a472e8c93c9a903831d8fe74ebfdb0363d1e19"} Sep 11 18:13:27.558542 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:27.558402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:28.166544 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:28.166509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" event={"ID":"c192bb39-b3dc-460c-9ff1-115899dcf9e7","Type":"ContainerStarted","Data":"802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1"} Sep 11 18:13:28.166919 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:28.166659 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:28.187462 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:28.187397 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" podStartSLOduration=2.670183761 podStartE2EDuration="3.187381176s" totalImagesPullingTime="517.197415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:28.185824775 +0000 UTC m=+1152.216886022" watchObservedRunningTime="2026-09-11 18:13:28.187381176 +0000 UTC m=+1152.218442405" Sep 11 18:13:28.322369 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:28.322331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:29.160786 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:29.160755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:29.880722 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:29.880692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:30.546853 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:30.546818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:32.470782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:32.470753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:32.575035 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:32.574999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:32.742993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:32.742909 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq"] Sep 11 18:13:32.743167 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:32.743141 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" containerID="cri-o://e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47" gracePeriod=30 Sep 11 18:13:32.768971 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:32.768945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:33.281927 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:33.281894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:33.617392 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:33.617344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:33.823017 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:33.822984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:34.176911 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:34.176885 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:13:35.525036 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:35.525005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:35.625466 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:35.625434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:35.730577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:35.730540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:36.048303 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:36.048266 2579 patch_prober.go:28] interesting pod/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:13:36.048499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:36.048328 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:13:38.002931 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.002907 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:13:38.072669 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072559 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-envoy\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072669 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072614 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istiod-ca-cert\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072674 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-certs\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072704 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59rzl\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-kube-api-access-59rzl\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-credential-socket\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-socket\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072813 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-data\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.072908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-podinfo\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.073191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.072940 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-token\") pod \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\" (UID: \"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5\") " Sep 11 18:13:38.073725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.073688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:38.073873 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.073845 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:38.073961 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.073938 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:38.074241 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.074217 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:38.074587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.074562 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:13:38.076007 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.075983 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:13:38.076443 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.076396 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-kube-api-access-59rzl" (OuterVolumeSpecName: "kube-api-access-59rzl") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "kube-api-access-59rzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:13:38.076923 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.076898 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:13:38.077299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.077269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" (UID: "93b9ff1c-e71d-4251-99c5-6778e4f4f2b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:13:38.174148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174113 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174140 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174148 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174158 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174166 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rzl\" (UniqueName: \"kubernetes.io/projected/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-kube-api-access-59rzl\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174178 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174186 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174195 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.174384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.174203 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:13:38.220034 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.219996 2579 generic.go:356] "Generic (PLEG): container finished" podID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerID="e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47" exitCode=0 Sep 11 18:13:38.220034 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.220029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" event={"ID":"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5","Type":"ContainerDied","Data":"e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47"} Sep 11 18:13:38.220222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.220051 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" Sep 11 18:13:38.220222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.220082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq" event={"ID":"93b9ff1c-e71d-4251-99c5-6778e4f4f2b5","Type":"ContainerDied","Data":"0458fe5c7882b9fc0b0ee39df5f46b79431243968bd0924c5cc8480403454ee5"} Sep 11 18:13:38.220222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.220105 2579 scope.go:117] "RemoveContainer" containerID="e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47" Sep 11 18:13:38.230076 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.230056 2579 scope.go:117] "RemoveContainer" containerID="200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099" Sep 11 18:13:38.240710 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.240693 2579 scope.go:117] "RemoveContainer" containerID="e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47" Sep 11 18:13:38.241013 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:13:38.240989 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47\": container with ID starting with e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47 not found: ID does not exist" containerID="e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47" Sep 11 18:13:38.241068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.241027 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47"} err="failed to get container status \"e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47\": rpc error: code = NotFound desc = could not find container \"e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47\": container with ID starting with e162828cebafb312e9e5c26a151e75ffbf216a01c375accee46c7ad864709e47 not found: ID does not exist" Sep 11 18:13:38.241068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.241052 2579 scope.go:117] "RemoveContainer" containerID="200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099" Sep 11 18:13:38.241365 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:13:38.241340 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099\": container with ID starting with 200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099 not found: ID does not exist" containerID="200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099" Sep 11 18:13:38.241443 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.241376 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099"} err="failed to get container status \"200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099\": rpc error: code = NotFound desc = could not find container \"200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099\": container with ID starting with 200ef079721572f480e5e4d6ff5ba5f67e456ab478ba39971b62a93ce82aa099 not found: ID does not exist" Sep 11 18:13:38.255565 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.255543 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq"] Sep 11 18:13:38.283077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.283051 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-aedf1a-openshift-default-6dd478f644-vjndq"] Sep 11 18:13:38.444794 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:38.444763 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" path="/var/lib/kubelet/pods/93b9ff1c-e71d-4251-99c5-6778e4f4f2b5/volumes" Sep 11 18:13:48.067751 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:48.067715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:49.173308 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.173279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:49.280616 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.280585 2579 generic.go:356] "Generic (PLEG): container finished" podID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerID="5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6" exitCode=0 Sep 11 18:13:49.280792 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.280662 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" event={"ID":"17f30f33-f227-43bf-b0cf-bc0b2bb2080f","Type":"ContainerDied","Data":"5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6"} Sep 11 18:13:49.281216 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.281196 2579 scope.go:117] "RemoveContainer" containerID="5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6" Sep 11 18:13:49.282722 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.282687 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:13:49.282826 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.282781 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:13:49.282880 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:49.282825 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:13:50.089177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:50.089139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:50.123002 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:50.122973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:50.289218 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:50.289187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" event={"ID":"17f30f33-f227-43bf-b0cf-bc0b2bb2080f","Type":"ContainerStarted","Data":"743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570"} Sep 11 18:13:51.172674 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:51.172633 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:51.174242 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:51.174206 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 11 18:13:51.174386 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:51.174268 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 11 18:13:51.189822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:51.189772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:51.406465 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:51.406388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:51.735387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:51.735355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:52.080494 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:52.079948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:52.173442 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:52.172817 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 11 18:13:52.173442 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:52.172881 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 11 18:13:52.185245 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:52.185198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:52.383128 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:52.383073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:52.943744 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:52.943704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:53.172809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:53.172771 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 11 18:13:53.173089 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:53.173060 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 11 18:13:54.120582 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:54.120533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:54.172868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:54.172828 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 11 18:13:54.173066 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:54.172890 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 11 18:13:54.821224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:54.821186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:55.172659 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:55.172620 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 11 18:13:55.173259 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:55.173225 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 11 18:13:55.406309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:55.406269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:55.499813 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:55.499773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:55.877566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:55.877534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:56.176585 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:56.176494 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:56.177126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:56.176933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:56.177653 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:56.177634 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:13:57.074013 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:57.073970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:57.885946 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:57.885912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:58.756621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:58.756586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:13:59.671608 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:13:59.671576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:00.604347 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:00.604314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:00.702820 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:00.702784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:00.837605 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:00.837573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:01.137942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.137900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:01.240782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.240749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:01.277056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.277026 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7"] Sep 11 18:14:01.277748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.277724 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" Sep 11 18:14:01.277886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.277818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" Sep 11 18:14:01.277886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.277849 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" Sep 11 18:14:01.277886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.277857 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" Sep 11 18:14:01.278045 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.278002 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" Sep 11 18:14:01.278045 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.278014 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b9ff1c-e71d-4251-99c5-6778e4f4f2b5" containerName="istio-proxy" Sep 11 18:14:01.280817 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.280798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.284036 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.284017 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-90057a\"" Sep 11 18:14:01.292634 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.292606 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7"] Sep 11 18:14:01.335395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.335360 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx"] Sep 11 18:14:01.337875 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.337854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.341778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.341757 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-90057a-dockercfg-5gk4p\"" Sep 11 18:14:01.359530 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.359498 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx"] Sep 11 18:14:01.415768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.415677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wng\" (UniqueName: \"kubernetes.io/projected/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-kube-api-access-67wng\") pod \"payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.415768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.415731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.416006 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.415804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9d9cd6-e6e0-4219-8347-bed285b66db9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.416006 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.415995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7p8\" (UniqueName: \"kubernetes.io/projected/4e9d9cd6-e6e0-4219-8347-bed285b66db9-kube-api-access-bt7p8\") pod \"payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.517318 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.517272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7p8\" (UniqueName: \"kubernetes.io/projected/4e9d9cd6-e6e0-4219-8347-bed285b66db9-kube-api-access-bt7p8\") pod \"payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.517535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.517330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wng\" (UniqueName: \"kubernetes.io/projected/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-kube-api-access-67wng\") pod \"payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.517535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.517357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.517535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.517376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9d9cd6-e6e0-4219-8347-bed285b66db9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.518242 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.518221 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9d9cd6-e6e0-4219-8347-bed285b66db9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.518411 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.518388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.527302 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.527270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wng\" (UniqueName: \"kubernetes.io/projected/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-kube-api-access-67wng\") pod \"payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.527467 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.527362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7p8\" (UniqueName: \"kubernetes.io/projected/4e9d9cd6-e6e0-4219-8347-bed285b66db9-kube-api-access-bt7p8\") pod \"payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.593191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.593150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:01.656592 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.656250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:01.867733 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.867701 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7"] Sep 11 18:14:01.868898 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:14:01.868865 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9d9cd6_e6e0_4219_8347_bed285b66db9.slice/crio-d1c18453a46cf3dc9fc074fc7928125f9fc95c309ea9ca37f41c9e6ee91b7289 WatchSource:0}: Error finding container d1c18453a46cf3dc9fc074fc7928125f9fc95c309ea9ca37f41c9e6ee91b7289: Status 404 returned error can't find the container with id d1c18453a46cf3dc9fc074fc7928125f9fc95c309ea9ca37f41c9e6ee91b7289 Sep 11 18:14:01.949295 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:01.949265 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx"] Sep 11 18:14:01.949959 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:14:01.949938 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8acc2c4_c2d7_46fa_aeac_fbf6432cf661.slice/crio-b54837dad220c19a854d06f127611e0645b10f1a7a92215d7eb86d6ba0cd1b1f WatchSource:0}: Error finding container b54837dad220c19a854d06f127611e0645b10f1a7a92215d7eb86d6ba0cd1b1f: Status 404 returned error can't find the container with id b54837dad220c19a854d06f127611e0645b10f1a7a92215d7eb86d6ba0cd1b1f Sep 11 18:14:02.361599 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.361488 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" event={"ID":"4e9d9cd6-e6e0-4219-8347-bed285b66db9","Type":"ContainerStarted","Data":"c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74"} Sep 11 18:14:02.361599 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.361552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" event={"ID":"4e9d9cd6-e6e0-4219-8347-bed285b66db9","Type":"ContainerStarted","Data":"d1c18453a46cf3dc9fc074fc7928125f9fc95c309ea9ca37f41c9e6ee91b7289"} Sep 11 18:14:02.361857 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.361680 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:02.363289 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.363263 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" event={"ID":"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661","Type":"ContainerStarted","Data":"416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44"} Sep 11 18:14:02.363405 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.363303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" event={"ID":"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661","Type":"ContainerStarted","Data":"b54837dad220c19a854d06f127611e0645b10f1a7a92215d7eb86d6ba0cd1b1f"} Sep 11 18:14:02.363405 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.363339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:02.382668 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.382607 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" podStartSLOduration=1.382587853 podStartE2EDuration="1.382587853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:02.382226245 +0000 UTC m=+1186.413287502" watchObservedRunningTime="2026-09-11 18:14:02.382587853 +0000 UTC m=+1186.413649082" Sep 11 18:14:02.406456 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:02.406388 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" podStartSLOduration=1.4063695 podStartE2EDuration="1.4063695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:02.40411519 +0000 UTC m=+1186.435176446" watchObservedRunningTime="2026-09-11 18:14:02.4063695 +0000 UTC m=+1186.437430730" Sep 11 18:14:13.370201 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:13.370172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:14:13.370672 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:13.370243 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:14:22.902276 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:22.902239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:23.663527 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:23.663492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:24.570361 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:24.570327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:25.461700 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:25.461668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:25.559071 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:25.559038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:26.244260 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:26.244227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:27.180707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:27.180676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:27.925407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:27.925371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:34.879005 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:34.878970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:35.210942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:35.210829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:35.432106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:35.432073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:37.937753 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:37.937722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:39.263478 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:39.263444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:39.869773 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:39.869737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:40.838177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:40.838131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:41.758607 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:41.758559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:42.703735 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:42.703704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:42.803737 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:42.803703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:43.637798 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:43.637751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:44.564367 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:44.564331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:44.667169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:44.667128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:44.843545 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:44.843407 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:45.100949 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.100829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:45.201564 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.201515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:45.227095 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.227057 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597"] Sep 11 18:14:45.230744 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.230722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.234076 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.234060 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3a652320-dockercfg-pjsdq\"" Sep 11 18:14:45.234397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.234381 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3a652320\"" Sep 11 18:14:45.243030 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.243009 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597"] Sep 11 18:14:45.367244 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.367043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2648a8eb-4ef7-4205-a395-aae2171a0499-plugins-config-volume\") pod \"payload-processing-e2e-mt-3a652320-7bd648857c-v5597\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.367244 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.367153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gg9\" (UniqueName: \"kubernetes.io/projected/2648a8eb-4ef7-4205-a395-aae2171a0499-kube-api-access-g8gg9\") pod \"payload-processing-e2e-mt-3a652320-7bd648857c-v5597\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.468252 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.468213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2648a8eb-4ef7-4205-a395-aae2171a0499-plugins-config-volume\") pod \"payload-processing-e2e-mt-3a652320-7bd648857c-v5597\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.468454 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.468303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gg9\" (UniqueName: \"kubernetes.io/projected/2648a8eb-4ef7-4205-a395-aae2171a0499-kube-api-access-g8gg9\") pod \"payload-processing-e2e-mt-3a652320-7bd648857c-v5597\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.468947 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.468927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2648a8eb-4ef7-4205-a395-aae2171a0499-plugins-config-volume\") pod \"payload-processing-e2e-mt-3a652320-7bd648857c-v5597\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.478288 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.478260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gg9\" (UniqueName: \"kubernetes.io/projected/2648a8eb-4ef7-4205-a395-aae2171a0499-kube-api-access-g8gg9\") pod \"payload-processing-e2e-mt-3a652320-7bd648857c-v5597\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.541205 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.541167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:45.820132 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:45.820105 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597"] Sep 11 18:14:45.822331 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:14:45.822304 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2648a8eb_4ef7_4205_a395_aae2171a0499.slice/crio-275c4ad1e929d137df9972c960882aa4610a9b87468fd219e3a4b78d6c275372 WatchSource:0}: Error finding container 275c4ad1e929d137df9972c960882aa4610a9b87468fd219e3a4b78d6c275372: Status 404 returned error can't find the container with id 275c4ad1e929d137df9972c960882aa4610a9b87468fd219e3a4b78d6c275372 Sep 11 18:14:46.065385 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:46.065352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:14:46.592852 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:46.592809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" event={"ID":"2648a8eb-4ef7-4205-a395-aae2171a0499","Type":"ContainerStarted","Data":"6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98"} Sep 11 18:14:46.593018 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:46.592872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" event={"ID":"2648a8eb-4ef7-4205-a395-aae2171a0499","Type":"ContainerStarted","Data":"275c4ad1e929d137df9972c960882aa4610a9b87468fd219e3a4b78d6c275372"} Sep 11 18:14:46.593018 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:46.592904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:14:46.615063 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:46.615011 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" podStartSLOduration=1.614995579 podStartE2EDuration="1.614995579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:46.612965887 +0000 UTC m=+1230.644027143" watchObservedRunningTime="2026-09-11 18:14:46.614995579 +0000 UTC m=+1230.646056809" Sep 11 18:14:57.601249 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:14:57.601214 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:15:04.493588 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:04.493546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:05.691063 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:05.691027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:06.610997 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:06.610956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:07.485393 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:07.485364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:08.350748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:08.350712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:09.294923 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:09.294884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:10.177497 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:10.177466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:10.968152 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:10.968118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:11.830586 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:11.830555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:12.723717 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:12.723684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:13.681322 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:13.681290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:13.805361 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:13.805321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:14.004706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:14.004600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:14.237235 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:14.237201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:14.342191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:14.342116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:14.498348 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:14.498316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:15.479570 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:15.479535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:16.507377 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:16.507345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:17.080656 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:17.080622 2579 scope.go:117] "RemoveContainer" containerID="317b6c120c901d093bf7f23a30487a55c8c3027d507809011a32b688f1d3832e" Sep 11 18:15:17.630825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:17.630792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:17.806477 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:17.806444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:18.617164 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:18.617133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:19.421731 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.421700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:19.521216 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.521183 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8"] Sep 11 18:15:19.521774 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.521734 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" podUID="0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" containerName="maas-api" containerID="cri-o://ca3e170491c4d02165443795566e527e50c9c80634c046b1de91a9e6d8a4ac4c" gracePeriod=30 Sep 11 18:15:19.555650 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.555613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:19.673770 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.673480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:19.763133 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.763076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:19.775121 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.775088 2579 generic.go:356] "Generic (PLEG): container finished" podID="0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" containerID="ca3e170491c4d02165443795566e527e50c9c80634c046b1de91a9e6d8a4ac4c" exitCode=0 Sep 11 18:15:19.775304 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.775159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" event={"ID":"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142","Type":"ContainerDied","Data":"ca3e170491c4d02165443795566e527e50c9c80634c046b1de91a9e6d8a4ac4c"} Sep 11 18:15:19.806948 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.806924 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:15:19.865985 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.865946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:19.917705 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.917678 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c48cv\" (UniqueName: \"kubernetes.io/projected/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-kube-api-access-c48cv\") pod \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " Sep 11 18:15:19.917874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.917726 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-maas-api-tls\") pod \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " Sep 11 18:15:19.917874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.917751 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-openshift-service-ca\") pod \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " Sep 11 18:15:19.917874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.917783 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-metrics-certs\") pod \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\" (UID: \"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142\") " Sep 11 18:15:19.918149 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.918126 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" (UID: "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:15:19.920109 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.920081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" (UID: "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:15:19.920236 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.920141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" (UID: "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:15:19.920236 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.920160 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-kube-api-access-c48cv" (OuterVolumeSpecName: "kube-api-access-c48cv") pod "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" (UID: "0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142"). InnerVolumeSpecName "kube-api-access-c48cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:15:19.936788 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:19.936759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:20.018876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.018846 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c48cv\" (UniqueName: \"kubernetes.io/projected/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-kube-api-access-c48cv\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:15:20.018876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.018874 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-maas-api-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:15:20.018876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.018886 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-openshift-service-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:15:20.019125 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.018895 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:15:20.271508 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.271391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:20.781527 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.781494 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" Sep 11 18:15:20.781944 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.781529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8" event={"ID":"0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142","Type":"ContainerDied","Data":"bd166e0d3c112b803486a3399fbe3b05e4cc67acf6e33c736e1aa1383c01c0e6"} Sep 11 18:15:20.781944 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.781584 2579 scope.go:117] "RemoveContainer" containerID="ca3e170491c4d02165443795566e527e50c9c80634c046b1de91a9e6d8a4ac4c" Sep 11 18:15:20.815433 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.815400 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8"] Sep 11 18:15:20.823390 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:20.823367 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93598a-5b44c74d6-wdwd8"] Sep 11 18:15:21.117967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:21.117941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:21.991110 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:21.991077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:22.444260 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:22.444216 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" path="/var/lib/kubelet/pods/0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142/volumes" Sep 11 18:15:22.813575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:22.813490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:23.599545 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:23.599517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:24.486494 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:24.486464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:25.344475 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:25.344400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:25.441681 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:25.441639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:27.744896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:27.744860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:31.816784 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:31.816724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:33.420312 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:33.420271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:33.659446 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:33.659400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:41.405831 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:41.405789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:41.628766 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:41.628730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:41.706302 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:41.706215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:42.840196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:42.840164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:43.874917 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:43.874882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:44.757821 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:44.757788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:46.047620 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:46.047580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:47.246555 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:47.246517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:48.278644 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:48.278610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:49.181370 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:49.181340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:52.575151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.575108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:52.606687 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.606654 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9"] Sep 11 18:15:52.606923 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.606877 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" podUID="cf9b1b18-2540-459d-8807-32511ebb2d5d" containerName="payload-processing" containerID="cri-o://9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3" gracePeriod=30 Sep 11 18:15:52.675577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.675539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:52.777154 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.777122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:52.843311 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.843217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:52.872674 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.872646 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:15:52.941226 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.941193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbqc4\" (UniqueName: \"kubernetes.io/projected/cf9b1b18-2540-459d-8807-32511ebb2d5d-kube-api-access-jbqc4\") pod \"cf9b1b18-2540-459d-8807-32511ebb2d5d\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " Sep 11 18:15:52.941434 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.941328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b1b18-2540-459d-8807-32511ebb2d5d-plugins-config-volume\") pod \"cf9b1b18-2540-459d-8807-32511ebb2d5d\" (UID: \"cf9b1b18-2540-459d-8807-32511ebb2d5d\") " Sep 11 18:15:52.941813 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.941786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9b1b18-2540-459d-8807-32511ebb2d5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf9b1b18-2540-459d-8807-32511ebb2d5d" (UID: "cf9b1b18-2540-459d-8807-32511ebb2d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:15:52.943792 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.943760 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9b1b18-2540-459d-8807-32511ebb2d5d-kube-api-access-jbqc4" (OuterVolumeSpecName: "kube-api-access-jbqc4") pod "cf9b1b18-2540-459d-8807-32511ebb2d5d" (UID: "cf9b1b18-2540-459d-8807-32511ebb2d5d"). InnerVolumeSpecName "kube-api-access-jbqc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:15:52.946485 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.946454 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf9b1b18-2540-459d-8807-32511ebb2d5d" containerID="9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3" exitCode=0 Sep 11 18:15:52.946589 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.946504 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" Sep 11 18:15:52.946658 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.946638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" event={"ID":"cf9b1b18-2540-459d-8807-32511ebb2d5d","Type":"ContainerDied","Data":"9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3"} Sep 11 18:15:52.946720 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.946697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9" event={"ID":"cf9b1b18-2540-459d-8807-32511ebb2d5d","Type":"ContainerDied","Data":"1c30ccd6ceb4203562a064b8c7103af2461ab505dfe23f1986051cc19354b1de"} Sep 11 18:15:52.946776 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.946730 2579 scope.go:117] "RemoveContainer" containerID="9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3" Sep 11 18:15:52.964694 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.963897 2579 scope.go:117] "RemoveContainer" containerID="9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3" Sep 11 18:15:52.964694 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:15:52.964286 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3\": container with ID starting with 9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3 not found: ID does not exist" containerID="9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3" Sep 11 18:15:52.964694 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.964318 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3"} err="failed to get container status \"9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3\": rpc error: code = NotFound desc = could not find container \"9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3\": container with ID starting with 9e7c6a0a0a0608243cc73bdd4d18baafb1e70c0a143dcd57be9e6808c83cc7c3 not found: ID does not exist" Sep 11 18:15:52.979305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.979256 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9"] Sep 11 18:15:52.985319 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:52.985288 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d59a02-5949b5bbc7-drzn9"] Sep 11 18:15:53.042407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:53.042375 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9b1b18-2540-459d-8807-32511ebb2d5d-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:15:53.042407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:53.042403 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqc4\" (UniqueName: \"kubernetes.io/projected/cf9b1b18-2540-459d-8807-32511ebb2d5d-kube-api-access-jbqc4\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:15:53.217107 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:53.217071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:53.322743 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:53.322714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:54.154842 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:54.154807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:54.444374 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:54.444284 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9b1b18-2540-459d-8807-32511ebb2d5d" path="/var/lib/kubelet/pods/cf9b1b18-2540-459d-8807-32511ebb2d5d/volumes" Sep 11 18:15:55.052337 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:55.052302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:55.867808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:55.867776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:56.775179 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:56.775146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:57.768773 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:57.768738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:59.862759 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:59.862731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:15:59.962281 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:15:59.962242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:00.135820 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:00.135788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:00.702909 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:00.702879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:01.047905 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:01.047803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:01.206936 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:01.206889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:03.815154 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:03.815115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:04.360993 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:04.360952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:04.455450 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:04.455391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:04.774621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:04.774576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:05.772186 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:05.772158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:06.704295 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:06.704260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:07.724888 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:07.724839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:08.521817 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:08.521757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:09.028411 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:09.028356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:09.319526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:09.319398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:09.421536 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:09.421405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:09.679681 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:09.679649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:09.953065 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:09.952984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:10.061254 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:10.061227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:24.006065 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:24.006018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:31.799275 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:31.799236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:32.621466 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:32.621432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:33.487243 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:33.487208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:34.345566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:34.345538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:34.456468 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:34.456401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:34.777288 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:34.777256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:35.089017 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.088926 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9"] Sep 11 18:16:35.089482 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.089464 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" containerName="maas-api" Sep 11 18:16:35.089482 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.089483 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" containerName="maas-api" Sep 11 18:16:35.089663 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.089524 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9b1b18-2540-459d-8807-32511ebb2d5d" containerName="payload-processing" Sep 11 18:16:35.089663 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.089533 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9b1b18-2540-459d-8807-32511ebb2d5d" containerName="payload-processing" Sep 11 18:16:35.097131 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.097104 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9b1b18-2540-459d-8807-32511ebb2d5d" containerName="payload-processing" Sep 11 18:16:35.097411 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.097387 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3558a1-e6d0-482c-9d1e-8bf5fb9e2142" containerName="maas-api" Sep 11 18:16:35.102217 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.102198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.106154 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.106134 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5d35e7-openshift-default-dockercfg-tk747\"" Sep 11 18:16:35.118519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.118495 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9"] Sep 11 18:16:35.127525 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.127489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:35.146665 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-istio-token\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.146773 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce032c31-6548-4e93-ab60-1463b3b02095-istiod-ca-cert\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.146773 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-envoy\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.146937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-data\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.146937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-credential-socket\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.146937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce032c31-6548-4e93-ab60-1463b3b02095-istio-podinfo\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.147101 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.146972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-socket\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.147101 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.147047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-certs\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.147251 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.147121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqd2m\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-kube-api-access-xqd2m\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248335 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqd2m\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-kube-api-access-xqd2m\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248668 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-istio-token\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248668 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce032c31-6548-4e93-ab60-1463b3b02095-istiod-ca-cert\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248889 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-envoy\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248889 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-data\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248889 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-credential-socket\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.248889 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce032c31-6548-4e93-ab60-1463b3b02095-istio-podinfo\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-socket\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.248932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-certs\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249266 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.249226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-credential-socket\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249392 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.249365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-certs\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.249366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-data\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.249556 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-socket\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.249606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.249557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce032c31-6548-4e93-ab60-1463b3b02095-istiod-ca-cert\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.251389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.251368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-envoy\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.251868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.251845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce032c31-6548-4e93-ab60-1463b3b02095-istio-podinfo\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.269367 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.269338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-istio-token\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.269511 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.269400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqd2m\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-kube-api-access-xqd2m\") pod \"e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.316499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.316464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:35.413998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.413969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:35.415106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.415075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:35.700896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.700873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9"] Sep 11 18:16:35.703341 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:16:35.703311 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce032c31_6548_4e93_ab60_1463b3b02095.slice/crio-1e0817bd411eda6510bb3e6d06492d67bb3bc56c068314eb59579f249aae426e WatchSource:0}: Error finding container 1e0817bd411eda6510bb3e6d06492d67bb3bc56c068314eb59579f249aae426e: Status 404 returned error can't find the container with id 1e0817bd411eda6510bb3e6d06492d67bb3bc56c068314eb59579f249aae426e Sep 11 18:16:35.705809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.705777 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:16:35.705896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.705843 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:16:35.705896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:35.705872 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:16:36.173695 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.173663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" event={"ID":"ce032c31-6548-4e93-ab60-1463b3b02095","Type":"ContainerStarted","Data":"42b0a7d0d2d5e673568f9ab234ac315fcff20acda9e6fca261a9869cebf05f8e"} Sep 11 18:16:36.174078 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.173717 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" event={"ID":"ce032c31-6548-4e93-ab60-1463b3b02095","Type":"ContainerStarted","Data":"1e0817bd411eda6510bb3e6d06492d67bb3bc56c068314eb59579f249aae426e"} Sep 11 18:16:36.197342 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.197293 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podStartSLOduration=1.197276026 podStartE2EDuration="1.197276026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:36.196129857 +0000 UTC m=+1340.227191108" watchObservedRunningTime="2026-09-11 18:16:36.197276026 +0000 UTC m=+1340.228337256" Sep 11 18:16:36.269221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.269192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:36.414850 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.414808 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:36.416603 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.416574 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 11 18:16:36.416737 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:36.416626 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 11 18:16:37.234069 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:37.234038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:37.418797 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:37.418768 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:38.182349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:38.182310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:38.187671 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:38.187637 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:38.188866 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:38.188837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:16:38.267495 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:38.267462 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:39.366300 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:39.366270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:40.132197 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:40.132164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:40.356259 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:40.356227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:41.099457 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:41.099412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:42.066339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:42.066295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:42.865316 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:42.865283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:43.759109 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:43.759070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:44.694932 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:44.694899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:44.791768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:44.791736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:45.072895 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.072805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:45.280223 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.280192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:45.384093 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.384060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:16:45.398955 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.398921 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c"] Sep 11 18:16:45.402533 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.402503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.405683 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.405658 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5d35e7\"" Sep 11 18:16:45.405906 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.405890 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5d35e7-dockercfg-ps4b7\"" Sep 11 18:16:45.416194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.416164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c"] Sep 11 18:16:45.557872 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.557834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2b8\" (UniqueName: \"kubernetes.io/projected/71760337-fcd1-4cf7-bb83-f0b3992a467a-kube-api-access-ht2b8\") pod \"payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.558061 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.557900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71760337-fcd1-4cf7-bb83-f0b3992a467a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.659134 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.659039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2b8\" (UniqueName: \"kubernetes.io/projected/71760337-fcd1-4cf7-bb83-f0b3992a467a-kube-api-access-ht2b8\") pod \"payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.659134 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.659097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71760337-fcd1-4cf7-bb83-f0b3992a467a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.659763 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.659746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71760337-fcd1-4cf7-bb83-f0b3992a467a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.668954 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.668930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2b8\" (UniqueName: \"kubernetes.io/projected/71760337-fcd1-4cf7-bb83-f0b3992a467a-kube-api-access-ht2b8\") pod \"payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:45.715960 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:45.715929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:46.013146 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:46.013118 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c"] Sep 11 18:16:46.015477 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:16:46.015452 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71760337_fcd1_4cf7_bb83_f0b3992a467a.slice/crio-dae03677158c0c18f7102f513028c7bf923bc10e95ee83b5bd89f058279dd29c WatchSource:0}: Error finding container dae03677158c0c18f7102f513028c7bf923bc10e95ee83b5bd89f058279dd29c: Status 404 returned error can't find the container with id dae03677158c0c18f7102f513028c7bf923bc10e95ee83b5bd89f058279dd29c Sep 11 18:16:46.234042 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:46.234000 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" event={"ID":"71760337-fcd1-4cf7-bb83-f0b3992a467a","Type":"ContainerStarted","Data":"00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b"} Sep 11 18:16:46.234239 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:46.234090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" event={"ID":"71760337-fcd1-4cf7-bb83-f0b3992a467a","Type":"ContainerStarted","Data":"dae03677158c0c18f7102f513028c7bf923bc10e95ee83b5bd89f058279dd29c"} Sep 11 18:16:46.234239 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:46.234224 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:16:46.255748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:46.255697 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" podStartSLOduration=1.255682102 podStartE2EDuration="1.255682102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:46.254391928 +0000 UTC m=+1350.285453180" watchObservedRunningTime="2026-09-11 18:16:46.255682102 +0000 UTC m=+1350.286743360" Sep 11 18:16:57.241347 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:16:57.241310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:17:05.332891 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:05.332787 2579 generic.go:356] "Generic (PLEG): container finished" podID="ce032c31-6548-4e93-ab60-1463b3b02095" containerID="42b0a7d0d2d5e673568f9ab234ac315fcff20acda9e6fca261a9869cebf05f8e" exitCode=0 Sep 11 18:17:05.332891 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:05.332861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" event={"ID":"ce032c31-6548-4e93-ab60-1463b3b02095","Type":"ContainerDied","Data":"42b0a7d0d2d5e673568f9ab234ac315fcff20acda9e6fca261a9869cebf05f8e"} Sep 11 18:17:05.333396 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:05.333288 2579 scope.go:117] "RemoveContainer" containerID="42b0a7d0d2d5e673568f9ab234ac315fcff20acda9e6fca261a9869cebf05f8e" Sep 11 18:17:05.334506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:05.334473 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:17:05.334605 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:05.334540 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:17:05.334605 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:05.334571 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:17:06.341002 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:06.340969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" event={"ID":"ce032c31-6548-4e93-ab60-1463b3b02095","Type":"ContainerStarted","Data":"05825e3cf3c1e50a04b5af0fc374410b8707761ce2f8b9dc2a3bf6c73ae483cf"} Sep 11 18:17:06.415027 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:06.414984 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:17:06.416161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:06.416133 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 11 18:17:06.416290 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:06.416181 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 11 18:17:07.381827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:07.381789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:07.414969 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:07.414938 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 11 18:17:07.415144 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:07.414995 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 11 18:17:08.188038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:08.188001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:17:08.228330 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:08.228299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:08.393124 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:08.393088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:08.415343 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:08.415312 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 11 18:17:08.415531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:08.415367 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 11 18:17:09.414964 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:09.414929 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 11 18:17:09.415343 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:09.414987 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 11 18:17:09.532591 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:09.532549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:10.414621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:10.414585 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 11 18:17:10.414791 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:10.414658 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 11 18:17:10.585448 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:10.585391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:11.416094 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:11.416035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:11.589222 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:11.589190 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:17:11.592219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:11.592191 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:17:12.470822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:12.470789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:13.256806 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:13.256765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:14.060959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:14.060923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:15.719029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:15.718995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:18.538782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:18.538744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:26.587485 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:26.587448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:27.520387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:27.520353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:28.489944 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:28.489908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:29.529956 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:29.529923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:30.427721 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:30.427685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:31.310584 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:31.310555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:32.161581 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:32.161544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:33.173362 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:33.173316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:33.973139 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:33.973100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:40.517895 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:40.517861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:42.102674 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.102637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:42.188302 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.188265 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597"] Sep 11 18:17:42.188534 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.188509 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" podUID="2648a8eb-4ef7-4205-a395-aae2171a0499" containerName="payload-processing" containerID="cri-o://6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98" gracePeriod=30 Sep 11 18:17:42.239408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.239370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:42.324092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.324040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:42.431006 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.430975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:42.456114 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.456091 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:17:42.532077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.532046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:42.546100 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.546066 2579 generic.go:356] "Generic (PLEG): container finished" podID="2648a8eb-4ef7-4205-a395-aae2171a0499" containerID="6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98" exitCode=0 Sep 11 18:17:42.546296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.546174 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" Sep 11 18:17:42.546296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.546202 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" event={"ID":"2648a8eb-4ef7-4205-a395-aae2171a0499","Type":"ContainerDied","Data":"6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98"} Sep 11 18:17:42.546296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.546288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597" event={"ID":"2648a8eb-4ef7-4205-a395-aae2171a0499","Type":"ContainerDied","Data":"275c4ad1e929d137df9972c960882aa4610a9b87468fd219e3a4b78d6c275372"} Sep 11 18:17:42.546478 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.546336 2579 scope.go:117] "RemoveContainer" containerID="6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98" Sep 11 18:17:42.549983 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.549953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8gg9\" (UniqueName: \"kubernetes.io/projected/2648a8eb-4ef7-4205-a395-aae2171a0499-kube-api-access-g8gg9\") pod \"2648a8eb-4ef7-4205-a395-aae2171a0499\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " Sep 11 18:17:42.550098 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.549989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2648a8eb-4ef7-4205-a395-aae2171a0499-plugins-config-volume\") pod \"2648a8eb-4ef7-4205-a395-aae2171a0499\" (UID: \"2648a8eb-4ef7-4205-a395-aae2171a0499\") " Sep 11 18:17:42.550691 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.550664 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2648a8eb-4ef7-4205-a395-aae2171a0499-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2648a8eb-4ef7-4205-a395-aae2171a0499" (UID: "2648a8eb-4ef7-4205-a395-aae2171a0499"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:17:42.552707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.552686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2648a8eb-4ef7-4205-a395-aae2171a0499-kube-api-access-g8gg9" (OuterVolumeSpecName: "kube-api-access-g8gg9") pod "2648a8eb-4ef7-4205-a395-aae2171a0499" (UID: "2648a8eb-4ef7-4205-a395-aae2171a0499"). InnerVolumeSpecName "kube-api-access-g8gg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:17:42.568539 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.568497 2579 scope.go:117] "RemoveContainer" containerID="6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98" Sep 11 18:17:42.568932 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:17:42.568894 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98\": container with ID starting with 6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98 not found: ID does not exist" containerID="6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98" Sep 11 18:17:42.569030 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.568944 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98"} err="failed to get container status \"6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98\": rpc error: code = NotFound desc = could not find container \"6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98\": container with ID starting with 6865a329c12c0480c78e1545328b33299492743d9e399f1fa223b4dc8160dd98 not found: ID does not exist" Sep 11 18:17:42.651770 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.651736 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8gg9\" (UniqueName: \"kubernetes.io/projected/2648a8eb-4ef7-4205-a395-aae2171a0499-kube-api-access-g8gg9\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:17:42.651770 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.651768 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2648a8eb-4ef7-4205-a395-aae2171a0499-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:17:42.876519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.876487 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597"] Sep 11 18:17:42.879725 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.879693 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a652320-7bd648857c-v5597"] Sep 11 18:17:42.984227 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:42.984135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:43.689743 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:43.689708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:43.788930 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:43.788890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:43.990963 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:43.990876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:44.445093 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:44.445063 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2648a8eb-4ef7-4205-a395-aae2171a0499" path="/var/lib/kubelet/pods/2648a8eb-4ef7-4205-a395-aae2171a0499/volumes" Sep 11 18:17:44.983542 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:44.983508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:45.702089 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:45.702060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:46.013690 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:46.013585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:46.844486 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:46.844453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:47.917357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:47.917322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:48.708715 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:48.708677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:50.781924 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:50.781894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:53.810748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:53.810712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:54.097114 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:54.097026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:17:58.242068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:17:58.242029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:03.242398 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:03.242368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:27.745408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:27.745377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:28.710642 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:28.710602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:33.944264 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:33.944225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:35.021818 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:35.021788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:41.284238 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:41.284188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:41.382843 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:41.382811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:41.591309 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:41.591220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:42.910611 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:42.910573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:43.008894 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:43.008859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:43.109590 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:43.109539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:43.739983 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:43.739942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:43.836556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:43.836518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:43.936293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:43.936262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:44.957150 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:44.957116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:45.066201 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:45.066170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:45.169132 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:45.169103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:47.361888 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:47.361850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:52.021383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:52.021346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:52.121620 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:52.121583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:52.691297 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:52.691266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:59.109099 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:59.109069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:59.183574 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:59.183534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:59.295169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:59.295134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:18:59.385893 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:18:59.385863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:09.808015 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:09.807978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:10.040106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:10.040066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:18.550043 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:18.550008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:22.372429 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:22.372383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:32.039763 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:32.039719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:39.243823 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:39.243735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:40.984472 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:40.984435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:42.231155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:42.231123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:42.971001 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:42.970968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:43.063914 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:43.063887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:43.165797 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:43.165768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:44.269794 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:44.269753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:45.142880 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:45.142847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:46.179742 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:46.179701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:47.300051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:47.299992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:48.352868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:48.352835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:49.363409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:49.363371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:50.361897 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:50.361871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:50.453355 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:50.453322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:50.559987 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:50.559952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:51.022078 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.022046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:51.117745 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.117713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:19:51.222441 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.218344 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql"] Sep 11 18:19:51.222441 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.219329 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2648a8eb-4ef7-4205-a395-aae2171a0499" containerName="payload-processing" Sep 11 18:19:51.222441 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.219348 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2648a8eb-4ef7-4205-a395-aae2171a0499" containerName="payload-processing" Sep 11 18:19:51.222441 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.219634 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2648a8eb-4ef7-4205-a395-aae2171a0499" containerName="payload-processing" Sep 11 18:19:51.223139 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.223110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.226650 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.226625 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-684bbd-dockercfg-ptshq\"" Sep 11 18:19:51.226795 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.226622 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-684bbd\"" Sep 11 18:19:51.234257 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.234234 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql"] Sep 11 18:19:51.351376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.351282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.351376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.351349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5fn\" (UniqueName: \"kubernetes.io/projected/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-kube-api-access-nv5fn\") pod \"payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.452820 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.452762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.453010 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.452846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5fn\" (UniqueName: \"kubernetes.io/projected/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-kube-api-access-nv5fn\") pod \"payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.453580 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.453561 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.463867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.463836 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5fn\" (UniqueName: \"kubernetes.io/projected/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-kube-api-access-nv5fn\") pod \"payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.535566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.535527 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:51.807681 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:51.807650 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql"] Sep 11 18:19:51.808898 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:19:51.808876 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700f316d_1ed1_4d5e_acd4_2e938f2a47a0.slice/crio-752ee9f3550ae7aefa444e6f5c7675d1a0afecb215ac3832dad0a58f2781cf49 WatchSource:0}: Error finding container 752ee9f3550ae7aefa444e6f5c7675d1a0afecb215ac3832dad0a58f2781cf49: Status 404 returned error can't find the container with id 752ee9f3550ae7aefa444e6f5c7675d1a0afecb215ac3832dad0a58f2781cf49 Sep 11 18:19:52.219561 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:52.219516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" event={"ID":"700f316d-1ed1-4d5e-acd4-2e938f2a47a0","Type":"ContainerStarted","Data":"ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f"} Sep 11 18:19:52.219771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:52.219588 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" event={"ID":"700f316d-1ed1-4d5e-acd4-2e938f2a47a0","Type":"ContainerStarted","Data":"752ee9f3550ae7aefa444e6f5c7675d1a0afecb215ac3832dad0a58f2781cf49"} Sep 11 18:19:52.219771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:52.219688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:19:52.238834 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:52.238786 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" podStartSLOduration=1.238770798 podStartE2EDuration="1.238770798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:52.237791555 +0000 UTC m=+1536.268852981" watchObservedRunningTime="2026-09-11 18:19:52.238770798 +0000 UTC m=+1536.269832028" Sep 11 18:19:56.912257 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:19:56.912220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:03.227117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:03.227088 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:20:12.938627 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:12.938580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:13.037967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.037931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:13.415151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.415113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:13.709978 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.709888 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj"] Sep 11 18:20:13.712743 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.712720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.716821 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.716792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-794d1c-openshift-default-dockercfg-stltk\"" Sep 11 18:20:13.727557 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.727534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj"] Sep 11 18:20:13.741866 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.741836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:13.774711 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-socket\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.774896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-credential-socket\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.774896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-token\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.774896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdg5x\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-kube-api-access-fdg5x\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.774896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef14f41-9a89-4163-b099-9ed713c9ae3c-istiod-ca-cert\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.775065 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-certs\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.775065 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-data\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.775065 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.774994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-envoy\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.775065 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.775045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-podinfo\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.842815 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.842780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:13.876301 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-socket\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876301 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-credential-socket\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-token\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg5x\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-kube-api-access-fdg5x\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef14f41-9a89-4163-b099-9ed713c9ae3c-istiod-ca-cert\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-certs\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-data\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.876630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-envoy\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.877019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-podinfo\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.877019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-socket\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.877019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.876890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-certs\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.877171 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.877084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-data\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.877171 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.877160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef14f41-9a89-4163-b099-9ed713c9ae3c-istiod-ca-cert\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.877273 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.877255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-credential-socket\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.879692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.879662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-envoy\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.879933 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.879916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-podinfo\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.888532 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.888500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-token\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.893406 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.893377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdg5x\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-kube-api-access-fdg5x\") pod \"e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:13.941829 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:13.941767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:14.026765 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.026662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:14.306587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.306562 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj"] Sep 11 18:20:14.307923 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:20:14.307895 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef14f41_9a89_4163_b099_9ed713c9ae3c.slice/crio-f5dc0c204cc2524e15e9cd0eb56b9a12a748d4a84e87921438c3b363218cda57 WatchSource:0}: Error finding container f5dc0c204cc2524e15e9cd0eb56b9a12a748d4a84e87921438c3b363218cda57: Status 404 returned error can't find the container with id f5dc0c204cc2524e15e9cd0eb56b9a12a748d4a84e87921438c3b363218cda57 Sep 11 18:20:14.310339 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.310310 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:20:14.310403 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.310375 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:20:14.310454 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.310403 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:20:14.329255 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.329219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" event={"ID":"bef14f41-9a89-4163-b099-9ed713c9ae3c","Type":"ContainerStarted","Data":"f5dc0c204cc2524e15e9cd0eb56b9a12a748d4a84e87921438c3b363218cda57"} Sep 11 18:20:14.841706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:14.841674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:15.336550 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:15.336517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" event={"ID":"bef14f41-9a89-4163-b099-9ed713c9ae3c","Type":"ContainerStarted","Data":"291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc"} Sep 11 18:20:15.361771 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:15.361720 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podStartSLOduration=2.361703983 podStartE2EDuration="2.361703983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:15.35866574 +0000 UTC m=+1559.389726986" watchObservedRunningTime="2026-09-11 18:20:15.361703983 +0000 UTC m=+1559.392765212" Sep 11 18:20:15.924676 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:15.924637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:16.027275 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:16.027245 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:16.032774 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:16.032750 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:16.341548 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:16.341459 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:16.342495 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:16.342475 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:16.877675 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:16.877644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:17.442042 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:17.442000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:17.549707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:17.549670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:18.026636 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:18.026600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:19.198987 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:19.198949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:20.067137 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:20.067102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:20.861606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:20.861568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:20.962857 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:20.962811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:21.122832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.122803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:21.405501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.405396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:21.500846 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.500812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:21.510406 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.510375 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6"] Sep 11 18:20:21.512974 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.512954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.515962 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.515935 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-794d1c\"" Sep 11 18:20:21.525293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.525270 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6"] Sep 11 18:20:21.581919 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.581889 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s"] Sep 11 18:20:21.584591 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.584575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.587656 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.587637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-794d1c-dockercfg-kcjgj\"" Sep 11 18:20:21.595992 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.595972 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s"] Sep 11 18:20:21.665714 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.664936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a9b339-ff1f-4411-a993-b9675d71e284-plugins-config-volume\") pod \"payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.665714 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.665003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8ww\" (UniqueName: \"kubernetes.io/projected/a3a9b339-ff1f-4411-a993-b9675d71e284-kube-api-access-6q8ww\") pod \"payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.765956 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.765919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8ww\" (UniqueName: \"kubernetes.io/projected/a3a9b339-ff1f-4411-a993-b9675d71e284-kube-api-access-6q8ww\") pod \"payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.766133 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.765996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783cd5d7-fbda-4148-a98a-e593a63341c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.766184 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.766131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkwn\" (UniqueName: \"kubernetes.io/projected/783cd5d7-fbda-4148-a98a-e593a63341c0-kube-api-access-kwkwn\") pod \"payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.766233 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.766214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a9b339-ff1f-4411-a993-b9675d71e284-plugins-config-volume\") pod \"payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.766875 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.766852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a9b339-ff1f-4411-a993-b9675d71e284-plugins-config-volume\") pod \"payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.775336 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.775318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8ww\" (UniqueName: \"kubernetes.io/projected/a3a9b339-ff1f-4411-a993-b9675d71e284-kube-api-access-6q8ww\") pod \"payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.824536 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.824500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:21.854237 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.854211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:21.867848 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.867798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkwn\" (UniqueName: \"kubernetes.io/projected/783cd5d7-fbda-4148-a98a-e593a63341c0-kube-api-access-kwkwn\") pod \"payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.868264 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.867953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783cd5d7-fbda-4148-a98a-e593a63341c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.868948 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.868925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783cd5d7-fbda-4148-a98a-e593a63341c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.879115 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.879063 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkwn\" (UniqueName: \"kubernetes.io/projected/783cd5d7-fbda-4148-a98a-e593a63341c0-kube-api-access-kwkwn\") pod \"payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:21.896267 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:21.896233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:22.099248 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.099225 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6"] Sep 11 18:20:22.100939 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:20:22.100913 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a9b339_ff1f_4411_a993_b9675d71e284.slice/crio-a3b98675710e2810239de5dc85f5715f61a3316d1f27259d30d52cfd02bcf21c WatchSource:0}: Error finding container a3b98675710e2810239de5dc85f5715f61a3316d1f27259d30d52cfd02bcf21c: Status 404 returned error can't find the container with id a3b98675710e2810239de5dc85f5715f61a3316d1f27259d30d52cfd02bcf21c Sep 11 18:20:22.215323 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.215250 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s"] Sep 11 18:20:22.216237 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:20:22.216214 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783cd5d7_fbda_4148_a98a_e593a63341c0.slice/crio-2881b9e94d6566d51636ac95ba37c3f39603b0d8954809e3be2c18e953ce2bb2 WatchSource:0}: Error finding container 2881b9e94d6566d51636ac95ba37c3f39603b0d8954809e3be2c18e953ce2bb2: Status 404 returned error can't find the container with id 2881b9e94d6566d51636ac95ba37c3f39603b0d8954809e3be2c18e953ce2bb2 Sep 11 18:20:22.373148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.373103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" event={"ID":"a3a9b339-ff1f-4411-a993-b9675d71e284","Type":"ContainerStarted","Data":"eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9"} Sep 11 18:20:22.373330 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.373171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" event={"ID":"a3a9b339-ff1f-4411-a993-b9675d71e284","Type":"ContainerStarted","Data":"a3b98675710e2810239de5dc85f5715f61a3316d1f27259d30d52cfd02bcf21c"} Sep 11 18:20:22.373330 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.373203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:22.375024 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.374996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" event={"ID":"783cd5d7-fbda-4148-a98a-e593a63341c0","Type":"ContainerStarted","Data":"4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793"} Sep 11 18:20:22.375160 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.375046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" event={"ID":"783cd5d7-fbda-4148-a98a-e593a63341c0","Type":"ContainerStarted","Data":"2881b9e94d6566d51636ac95ba37c3f39603b0d8954809e3be2c18e953ce2bb2"} Sep 11 18:20:22.375160 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.375086 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:22.402084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.402032 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" podStartSLOduration=1.402015234 podStartE2EDuration="1.402015234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:22.399924246 +0000 UTC m=+1566.430985493" watchObservedRunningTime="2026-09-11 18:20:22.402015234 +0000 UTC m=+1566.433076461" Sep 11 18:20:22.419772 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.419721 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" podStartSLOduration=1.419705754 podStartE2EDuration="1.419705754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:22.41880505 +0000 UTC m=+1566.449866295" watchObservedRunningTime="2026-09-11 18:20:22.419705754 +0000 UTC m=+1566.450767018" Sep 11 18:20:22.984477 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:22.984405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:23.863906 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:23.863874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:24.775827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:24.775791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:25.561671 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:25.561638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:33.382604 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:33.382572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:20:33.382987 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:33.382746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:20:39.303375 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:39.303342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:43.482945 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:43.482901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:44.480196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:44.480158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:45.389953 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:45.389920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:45.552876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:45.552837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:46.341935 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:46.341850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:47.261465 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:47.261434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:47.382636 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:47.382592 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 18:20:47.382804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:47.382676 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 18:20:47.394484 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:47.394385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:47.976074 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:47.976035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:48.841128 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:48.841101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:49.537636 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.537593 2579 generic.go:356] "Generic (PLEG): container finished" podID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerID="291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc" exitCode=0 Sep 11 18:20:49.537801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.537665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" event={"ID":"bef14f41-9a89-4163-b099-9ed713c9ae3c","Type":"ContainerDied","Data":"291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc"} Sep 11 18:20:49.538130 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.538115 2579 scope.go:117] "RemoveContainer" containerID="291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc" Sep 11 18:20:49.539457 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.539382 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:20:49.539573 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.539479 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:20:49.539573 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.539514 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:20:49.849258 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:49.849177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:50.545197 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:50.545165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" event={"ID":"bef14f41-9a89-4163-b099-9ed713c9ae3c","Type":"ContainerStarted","Data":"d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7"} Sep 11 18:20:50.667331 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:50.667292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:50.759625 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:50.759582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:50.861531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:50.861445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:51.027787 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:51.027756 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:51.028030 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:51.027999 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 11 18:20:51.028166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:51.028064 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 11 18:20:51.984647 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:51.984602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:52.027519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:52.027479 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 11 18:20:52.027722 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:52.027553 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 11 18:20:53.016663 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:53.016625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:53.027609 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:53.027579 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 11 18:20:53.027790 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:53.027656 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 11 18:20:53.879682 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:53.879640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:54.027717 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:54.027682 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 11 18:20:54.028082 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:54.027750 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 11 18:20:54.869327 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:54.869285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:55.027639 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:55.027601 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 11 18:20:55.027821 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:55.027655 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 11 18:20:55.759351 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:55.759240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:56.027909 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:56.027825 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 11 18:20:56.027909 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:56.027892 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 11 18:20:56.679930 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:56.679896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:57.031095 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:57.031008 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:57.031565 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:57.031442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:57.032117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:57.032097 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:20:57.480619 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:57.480580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:20:58.191594 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:20:58.191558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:00.475761 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:00.475728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:01.365054 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.365017 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp"] Sep 11 18:21:01.365272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.365232 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" podUID="4614eb16-3474-45ce-a151-51bc8c239020" containerName="maas-api" containerID="cri-o://fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661" gracePeriod=30 Sep 11 18:21:01.409756 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.409707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:01.513412 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.513372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:01.609058 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.609026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:01.697119 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.696877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:01.843729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.843700 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:21:01.894757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.894656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-metrics-certs\") pod \"4614eb16-3474-45ce-a151-51bc8c239020\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " Sep 11 18:21:01.894757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.894727 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ccph\" (UniqueName: \"kubernetes.io/projected/4614eb16-3474-45ce-a151-51bc8c239020-kube-api-access-7ccph\") pod \"4614eb16-3474-45ce-a151-51bc8c239020\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " Sep 11 18:21:01.894757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.894758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4614eb16-3474-45ce-a151-51bc8c239020-openshift-service-ca\") pod \"4614eb16-3474-45ce-a151-51bc8c239020\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " Sep 11 18:21:01.895049 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.894774 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-maas-api-tls\") pod \"4614eb16-3474-45ce-a151-51bc8c239020\" (UID: \"4614eb16-3474-45ce-a151-51bc8c239020\") " Sep 11 18:21:01.895218 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.895186 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4614eb16-3474-45ce-a151-51bc8c239020-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4614eb16-3474-45ce-a151-51bc8c239020" (UID: "4614eb16-3474-45ce-a151-51bc8c239020"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:01.897592 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.897552 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4614eb16-3474-45ce-a151-51bc8c239020" (UID: "4614eb16-3474-45ce-a151-51bc8c239020"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:21:01.897592 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.897576 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4614eb16-3474-45ce-a151-51bc8c239020" (UID: "4614eb16-3474-45ce-a151-51bc8c239020"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:21:01.897841 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.897819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4614eb16-3474-45ce-a151-51bc8c239020-kube-api-access-7ccph" (OuterVolumeSpecName: "kube-api-access-7ccph") pod "4614eb16-3474-45ce-a151-51bc8c239020" (UID: "4614eb16-3474-45ce-a151-51bc8c239020"). InnerVolumeSpecName "kube-api-access-7ccph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:01.946030 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.946001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:02.003079 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:01.996326 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:02.003272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.003095 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ccph\" (UniqueName: \"kubernetes.io/projected/4614eb16-3474-45ce-a151-51bc8c239020-kube-api-access-7ccph\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:02.003272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.003147 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4614eb16-3474-45ce-a151-51bc8c239020-openshift-service-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:02.003272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.003169 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4614eb16-3474-45ce-a151-51bc8c239020-maas-api-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:02.157241 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.157147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:02.624037 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.624000 2579 generic.go:356] "Generic (PLEG): container finished" podID="4614eb16-3474-45ce-a151-51bc8c239020" containerID="fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661" exitCode=0 Sep 11 18:21:02.624518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.624055 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" Sep 11 18:21:02.624518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.624088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" event={"ID":"4614eb16-3474-45ce-a151-51bc8c239020","Type":"ContainerDied","Data":"fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661"} Sep 11 18:21:02.624518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.624145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp" event={"ID":"4614eb16-3474-45ce-a151-51bc8c239020","Type":"ContainerDied","Data":"96569170f39ee994451ba575c16350c6b1cdf1bc74c75ca210cdafeecb214f20"} Sep 11 18:21:02.624518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.624171 2579 scope.go:117] "RemoveContainer" containerID="fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661" Sep 11 18:21:02.635236 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.635161 2579 scope.go:117] "RemoveContainer" containerID="fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661" Sep 11 18:21:02.635864 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:02.635842 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661\": container with ID starting with fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661 not found: ID does not exist" containerID="fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661" Sep 11 18:21:02.635984 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.635871 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661"} err="failed to get container status \"fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661\": rpc error: code = NotFound desc = could not find container \"fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661\": container with ID starting with fd87b3977079b0d48790cb813454a38681bbcc6f08718857a4824f9d21610661 not found: ID does not exist" Sep 11 18:21:02.644110 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.644087 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp"] Sep 11 18:21:02.651041 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:02.651016 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f8a2d4-5c49d57d69-sq7vp"] Sep 11 18:21:03.017606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:03.017573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:03.767514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:03.767481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:04.447097 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:04.447062 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4614eb16-3474-45ce-a151-51bc8c239020" path="/var/lib/kubelet/pods/4614eb16-3474-45ce-a151-51bc8c239020/volumes" Sep 11 18:21:04.574435 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:04.574396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:05.444573 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:05.444519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:06.152429 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:06.152396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:07.184767 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:07.184736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:07.795031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:07.794990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:07.838216 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:07.838167 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql"] Sep 11 18:21:07.839105 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:07.839057 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" podUID="700f316d-1ed1-4d5e-acd4-2e938f2a47a0" containerName="payload-pre-processing" containerID="cri-o://ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f" gracePeriod=30 Sep 11 18:21:07.896179 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:07.896140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:08.001288 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.001259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:08.096553 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.096528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:08.102506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.102485 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:21:08.177314 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.176290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-plugins-config-volume\") pod \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " Sep 11 18:21:08.177314 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.176348 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv5fn\" (UniqueName: \"kubernetes.io/projected/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-kube-api-access-nv5fn\") pod \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\" (UID: \"700f316d-1ed1-4d5e-acd4-2e938f2a47a0\") " Sep 11 18:21:08.177828 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.177762 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "700f316d-1ed1-4d5e-acd4-2e938f2a47a0" (UID: "700f316d-1ed1-4d5e-acd4-2e938f2a47a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:08.193136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.193075 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-kube-api-access-nv5fn" (OuterVolumeSpecName: "kube-api-access-nv5fn") pod "700f316d-1ed1-4d5e-acd4-2e938f2a47a0" (UID: "700f316d-1ed1-4d5e-acd4-2e938f2a47a0"). InnerVolumeSpecName "kube-api-access-nv5fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:08.277859 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.277820 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:08.277859 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.277855 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv5fn\" (UniqueName: \"kubernetes.io/projected/700f316d-1ed1-4d5e-acd4-2e938f2a47a0-kube-api-access-nv5fn\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:08.416670 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.416634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:08.504213 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.504164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:08.659635 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.659603 2579 generic.go:356] "Generic (PLEG): container finished" podID="700f316d-1ed1-4d5e-acd4-2e938f2a47a0" containerID="ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f" exitCode=0 Sep 11 18:21:08.659833 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.659648 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" Sep 11 18:21:08.659833 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.659679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" event={"ID":"700f316d-1ed1-4d5e-acd4-2e938f2a47a0","Type":"ContainerDied","Data":"ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f"} Sep 11 18:21:08.659833 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.659742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql" event={"ID":"700f316d-1ed1-4d5e-acd4-2e938f2a47a0","Type":"ContainerDied","Data":"752ee9f3550ae7aefa444e6f5c7675d1a0afecb215ac3832dad0a58f2781cf49"} Sep 11 18:21:08.659833 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.659775 2579 scope.go:117] "RemoveContainer" containerID="ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f" Sep 11 18:21:08.669272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.669250 2579 scope.go:117] "RemoveContainer" containerID="ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f" Sep 11 18:21:08.669585 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:08.669566 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f\": container with ID starting with ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f not found: ID does not exist" containerID="ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f" Sep 11 18:21:08.669655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.669593 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f"} err="failed to get container status \"ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f\": rpc error: code = NotFound desc = could not find container \"ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f\": container with ID starting with ed168a99a2e7893c29aa7dd852c116c4e2c72003655b5cbd1c6da1d827bc0c8f not found: ID does not exist" Sep 11 18:21:08.689254 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.689221 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql"] Sep 11 18:21:08.695633 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:08.695606 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-684bbd-7767459458-r9gql"] Sep 11 18:21:09.449703 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:09.449651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:09.680352 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:09.680318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:09.799941 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:09.799856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:09.974107 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:09.974072 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m"] Sep 11 18:21:09.974293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:09.974268 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" containerID="cri-o://17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd" gracePeriod=30 Sep 11 18:21:09.976943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:09.976922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:10.375695 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:10.375664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:10.446243 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:10.446213 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="700f316d-1ed1-4d5e-acd4-2e938f2a47a0" path="/var/lib/kubelet/pods/700f316d-1ed1-4d5e-acd4-2e938f2a47a0/volumes" Sep 11 18:21:11.261794 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:11.261759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:12.012575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:12.012540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:12.918956 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:12.918923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:13.676115 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:13.676082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:14.763029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:14.762998 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:21:14.763373 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:14.763058 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:21:15.236714 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.236690 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:21:15.351074 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.350975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-istio-token\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351074 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351035 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-envoy\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351074 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351072 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65873c38-8f3d-4261-972e-23c9b1d91664-istio-podinfo\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351294 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351227 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-socket\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351294 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-credential-socket\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351403 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351303 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-data\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351403 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-certs\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351553 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351409 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtt2x\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-kube-api-access-dtt2x\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351553 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65873c38-8f3d-4261-972e-23c9b1d91664-istiod-ca-cert\") pod \"65873c38-8f3d-4261-972e-23c9b1d91664\" (UID: \"65873c38-8f3d-4261-972e-23c9b1d91664\") " Sep 11 18:21:15.351553 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351533 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:15.351700 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351574 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:15.351700 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.351622 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:15.352125 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.352098 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.352262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.352129 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.352262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.352144 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.352262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.352102 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-data" (OuterVolumeSpecName: "istio-data") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:15.352262 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.352108 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65873c38-8f3d-4261-972e-23c9b1d91664-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:15.353393 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.353365 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65873c38-8f3d-4261-972e-23c9b1d91664-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:21:15.353541 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.353438 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-istio-token" (OuterVolumeSpecName: "istio-token") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:15.353541 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.353449 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:15.353886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.353871 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-kube-api-access-dtt2x" (OuterVolumeSpecName: "kube-api-access-dtt2x") pod "65873c38-8f3d-4261-972e-23c9b1d91664" (UID: "65873c38-8f3d-4261-972e-23c9b1d91664"). InnerVolumeSpecName "kube-api-access-dtt2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:15.452754 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.452721 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtt2x\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-kube-api-access-dtt2x\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.452754 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.452750 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65873c38-8f3d-4261-972e-23c9b1d91664-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.452754 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.452760 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65873c38-8f3d-4261-972e-23c9b1d91664-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.452979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.452768 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.452979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.452776 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65873c38-8f3d-4261-972e-23c9b1d91664-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.452979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.452784 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65873c38-8f3d-4261-972e-23c9b1d91664-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:15.671204 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.671173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:15.702624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.702594 2579 generic.go:356] "Generic (PLEG): container finished" podID="65873c38-8f3d-4261-972e-23c9b1d91664" containerID="17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd" exitCode=0 Sep 11 18:21:15.702801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.702649 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" Sep 11 18:21:15.702801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.702647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" event={"ID":"65873c38-8f3d-4261-972e-23c9b1d91664","Type":"ContainerDied","Data":"17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd"} Sep 11 18:21:15.702801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.702770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m" event={"ID":"65873c38-8f3d-4261-972e-23c9b1d91664","Type":"ContainerDied","Data":"91a225af9a1be08660714257d6f969403738f1135f71f8b8636690039ae9cf48"} Sep 11 18:21:15.702801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.702795 2579 scope.go:117] "RemoveContainer" containerID="17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd" Sep 11 18:21:15.713151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.713133 2579 scope.go:117] "RemoveContainer" containerID="1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725" Sep 11 18:21:15.721564 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.721547 2579 scope.go:117] "RemoveContainer" containerID="17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd" Sep 11 18:21:15.721856 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:15.721836 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd\": container with ID starting with 17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd not found: ID does not exist" containerID="17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd" Sep 11 18:21:15.721907 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.721863 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd"} err="failed to get container status \"17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd\": rpc error: code = NotFound desc = could not find container \"17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd\": container with ID starting with 17d7ac85a823798d97782b56a645d0cdd9a4d4eb1664bc6cae4a1d82247c0ddd not found: ID does not exist" Sep 11 18:21:15.721907 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.721881 2579 scope.go:117] "RemoveContainer" containerID="1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725" Sep 11 18:21:15.722142 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:15.722125 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725\": container with ID starting with 1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725 not found: ID does not exist" containerID="1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725" Sep 11 18:21:15.722203 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.722146 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725"} err="failed to get container status \"1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725\": rpc error: code = NotFound desc = could not find container \"1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725\": container with ID starting with 1e90dac90f8db7c8c0f06a3b8dfe794993087816cf7623578c2691b63585f725 not found: ID does not exist" Sep 11 18:21:15.728395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.728375 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m"] Sep 11 18:21:15.733575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.733550 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f8a2d4-openshift-default-76c675c9c8-hff6m"] Sep 11 18:21:15.738912 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.738880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:15.940274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:15.940193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:16.446466 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:16.446439 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" path="/var/lib/kubelet/pods/65873c38-8f3d-4261-972e-23c9b1d91664/volumes" Sep 11 18:21:16.538686 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:16.538659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:17.465556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:17.465464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:18.227240 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:18.227208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:19.090126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:19.090086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:19.856728 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:19.856695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:20.690916 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:20.690885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:21.568610 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:21.568579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:22.327224 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:22.327190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:27.620084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:27.620046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:27.718955 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:27.718927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:29.505120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:29.505081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:31.076410 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.076367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:31.879156 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.879122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:31.910992 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.910294 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6"] Sep 11 18:21:31.910992 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.910676 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" podUID="a3a9b339-ff1f-4411-a993-b9675d71e284" containerName="payload-processing" containerID="cri-o://eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9" gracePeriod=30 Sep 11 18:21:31.913943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.913913 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s"] Sep 11 18:21:31.914218 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.914157 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" podUID="783cd5d7-fbda-4148-a98a-e593a63341c0" containerName="payload-pre-processing" containerID="cri-o://4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793" gracePeriod=30 Sep 11 18:21:31.964525 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:31.964493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:32.036755 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.036722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:32.110921 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.110894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:32.208197 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.208175 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:21:32.211227 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.211195 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:21:32.332911 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.332877 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwkwn\" (UniqueName: \"kubernetes.io/projected/783cd5d7-fbda-4148-a98a-e593a63341c0-kube-api-access-kwkwn\") pod \"783cd5d7-fbda-4148-a98a-e593a63341c0\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " Sep 11 18:21:32.333111 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.332984 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a9b339-ff1f-4411-a993-b9675d71e284-plugins-config-volume\") pod \"a3a9b339-ff1f-4411-a993-b9675d71e284\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " Sep 11 18:21:32.333111 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.333017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q8ww\" (UniqueName: \"kubernetes.io/projected/a3a9b339-ff1f-4411-a993-b9675d71e284-kube-api-access-6q8ww\") pod \"a3a9b339-ff1f-4411-a993-b9675d71e284\" (UID: \"a3a9b339-ff1f-4411-a993-b9675d71e284\") " Sep 11 18:21:32.333111 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.333076 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783cd5d7-fbda-4148-a98a-e593a63341c0-plugins-config-volume\") pod \"783cd5d7-fbda-4148-a98a-e593a63341c0\" (UID: \"783cd5d7-fbda-4148-a98a-e593a63341c0\") " Sep 11 18:21:32.333386 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.333360 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a9b339-ff1f-4411-a993-b9675d71e284-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3a9b339-ff1f-4411-a993-b9675d71e284" (UID: "a3a9b339-ff1f-4411-a993-b9675d71e284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:32.333612 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.333589 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783cd5d7-fbda-4148-a98a-e593a63341c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "783cd5d7-fbda-4148-a98a-e593a63341c0" (UID: "783cd5d7-fbda-4148-a98a-e593a63341c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:32.335408 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.335383 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783cd5d7-fbda-4148-a98a-e593a63341c0-kube-api-access-kwkwn" (OuterVolumeSpecName: "kube-api-access-kwkwn") pod "783cd5d7-fbda-4148-a98a-e593a63341c0" (UID: "783cd5d7-fbda-4148-a98a-e593a63341c0"). InnerVolumeSpecName "kube-api-access-kwkwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:32.336272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.336250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a9b339-ff1f-4411-a993-b9675d71e284-kube-api-access-6q8ww" (OuterVolumeSpecName: "kube-api-access-6q8ww") pod "a3a9b339-ff1f-4411-a993-b9675d71e284" (UID: "a3a9b339-ff1f-4411-a993-b9675d71e284"). InnerVolumeSpecName "kube-api-access-6q8ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:32.433967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.433870 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/783cd5d7-fbda-4148-a98a-e593a63341c0-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:32.433967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.433903 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkwn\" (UniqueName: \"kubernetes.io/projected/783cd5d7-fbda-4148-a98a-e593a63341c0-kube-api-access-kwkwn\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:32.433967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.433916 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a9b339-ff1f-4411-a993-b9675d71e284-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:32.433967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.433927 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8ww\" (UniqueName: \"kubernetes.io/projected/a3a9b339-ff1f-4411-a993-b9675d71e284-kube-api-access-6q8ww\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:32.453200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.453173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:32.575636 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.575605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:32.794874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.794786 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3a9b339-ff1f-4411-a993-b9675d71e284" containerID="eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9" exitCode=0 Sep 11 18:21:32.794874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.794836 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" Sep 11 18:21:32.795105 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.794871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" event={"ID":"a3a9b339-ff1f-4411-a993-b9675d71e284","Type":"ContainerDied","Data":"eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9"} Sep 11 18:21:32.795105 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.794934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6" event={"ID":"a3a9b339-ff1f-4411-a993-b9675d71e284","Type":"ContainerDied","Data":"a3b98675710e2810239de5dc85f5715f61a3316d1f27259d30d52cfd02bcf21c"} Sep 11 18:21:32.795105 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.794976 2579 scope.go:117] "RemoveContainer" containerID="eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9" Sep 11 18:21:32.796370 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.796346 2579 generic.go:356] "Generic (PLEG): container finished" podID="783cd5d7-fbda-4148-a98a-e593a63341c0" containerID="4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793" exitCode=0 Sep 11 18:21:32.796506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.796404 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" Sep 11 18:21:32.796506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.796448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" event={"ID":"783cd5d7-fbda-4148-a98a-e593a63341c0","Type":"ContainerDied","Data":"4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793"} Sep 11 18:21:32.796506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.796488 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s" event={"ID":"783cd5d7-fbda-4148-a98a-e593a63341c0","Type":"ContainerDied","Data":"2881b9e94d6566d51636ac95ba37c3f39603b0d8954809e3be2c18e953ce2bb2"} Sep 11 18:21:32.804105 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.804091 2579 scope.go:117] "RemoveContainer" containerID="eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9" Sep 11 18:21:32.804347 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:32.804332 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9\": container with ID starting with eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9 not found: ID does not exist" containerID="eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9" Sep 11 18:21:32.804383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.804356 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9"} err="failed to get container status \"eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9\": rpc error: code = NotFound desc = could not find container \"eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9\": container with ID starting with eac28b921a11d9b48b73af01c115dd8c3422872562544ad654c45f849d4ee2a9 not found: ID does not exist" Sep 11 18:21:32.804383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.804371 2579 scope.go:117] "RemoveContainer" containerID="4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793" Sep 11 18:21:32.812301 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.812282 2579 scope.go:117] "RemoveContainer" containerID="4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793" Sep 11 18:21:32.812608 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:32.812589 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793\": container with ID starting with 4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793 not found: ID does not exist" containerID="4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793" Sep 11 18:21:32.812666 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.812615 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793"} err="failed to get container status \"4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793\": rpc error: code = NotFound desc = could not find container \"4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793\": container with ID starting with 4b8f6ec2e430708871aca89c448569798be1425e7aa9327fef49e21f72a80793 not found: ID does not exist" Sep 11 18:21:32.816179 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.816158 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s"] Sep 11 18:21:32.819840 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.819820 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-794d1c-7df8bd6c74-m5z9s"] Sep 11 18:21:32.831621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.831598 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6"] Sep 11 18:21:32.835728 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:32.835710 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-794d1c-f6b5cd69c-42cd6"] Sep 11 18:21:33.461817 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:33.461780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:34.153959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:34.153925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:34.444445 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:34.444342 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783cd5d7-fbda-4148-a98a-e593a63341c0" path="/var/lib/kubelet/pods/783cd5d7-fbda-4148-a98a-e593a63341c0/volumes" Sep 11 18:21:34.444748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:34.444734 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a9b339-ff1f-4411-a993-b9675d71e284" path="/var/lib/kubelet/pods/a3a9b339-ff1f-4411-a993-b9675d71e284/volumes" Sep 11 18:21:35.042931 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:35.042897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:35.758726 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:35.758698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:36.016965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:36.016879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:36.613643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:36.613603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:38.513832 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:38.513800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:38.613226 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:38.613185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:38.756686 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:38.756653 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj"] Sep 11 18:21:38.756928 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:38.756904 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" containerID="cri-o://d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7" gracePeriod=30 Sep 11 18:21:38.782085 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:38.781990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:39.450721 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:39.450694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:40.264629 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:40.264597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:41.085868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:41.085833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:41.722071 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:41.722015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:42.034123 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:42.034028 2579 patch_prober.go:28] interesting pod/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:21:42.034259 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:42.034115 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:21:42.556441 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:42.556386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:43.254002 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:43.253956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:43.965840 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:43.965805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:44.012015 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.011992 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:21:44.160055 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160018 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-data\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160255 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160073 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdg5x\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-kube-api-access-fdg5x\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160255 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160097 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-credential-socket\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160255 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160142 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-envoy\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160255 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef14f41-9a89-4163-b099-9ed713c9ae3c-istiod-ca-cert\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160521 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160295 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-socket\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160521 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-certs\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160521 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-podinfo\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160521 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160465 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-token\") pod \"bef14f41-9a89-4163-b099-9ed713c9ae3c\" (UID: \"bef14f41-9a89-4163-b099-9ed713c9ae3c\") " Sep 11 18:21:44.160803 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160396 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:44.160898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160489 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:44.160898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160700 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:44.160898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:44.160898 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160775 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef14f41-9a89-4163-b099-9ed713c9ae3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:44.161056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160954 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.161056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160973 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef14f41-9a89-4163-b099-9ed713c9ae3c-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.161056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160987 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.161056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.160999 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.161056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.161012 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.162743 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.162715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-kube-api-access-fdg5x" (OuterVolumeSpecName: "kube-api-access-fdg5x") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "kube-api-access-fdg5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:44.162874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.162778 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:21:44.163155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.163135 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:21:44.163221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.163190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "bef14f41-9a89-4163-b099-9ed713c9ae3c" (UID: "bef14f41-9a89-4163-b099-9ed713c9ae3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:44.261772 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.261732 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.261772 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.261770 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.261772 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.261779 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.262211 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.261788 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdg5x\" (UniqueName: \"kubernetes.io/projected/bef14f41-9a89-4163-b099-9ed713c9ae3c-kube-api-access-fdg5x\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:44.868201 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.868172 2579 generic.go:356] "Generic (PLEG): container finished" podID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerID="d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7" exitCode=0 Sep 11 18:21:44.868349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.868221 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" Sep 11 18:21:44.868349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.868237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" event={"ID":"bef14f41-9a89-4163-b099-9ed713c9ae3c","Type":"ContainerDied","Data":"d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7"} Sep 11 18:21:44.868349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.868281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj" event={"ID":"bef14f41-9a89-4163-b099-9ed713c9ae3c","Type":"ContainerDied","Data":"f5dc0c204cc2524e15e9cd0eb56b9a12a748d4a84e87921438c3b363218cda57"} Sep 11 18:21:44.868349 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.868318 2579 scope.go:117] "RemoveContainer" containerID="d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7" Sep 11 18:21:44.877645 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.877617 2579 scope.go:117] "RemoveContainer" containerID="291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc" Sep 11 18:21:44.887400 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.887381 2579 scope.go:117] "RemoveContainer" containerID="d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7" Sep 11 18:21:44.887800 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:44.887779 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7\": container with ID starting with d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7 not found: ID does not exist" containerID="d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7" Sep 11 18:21:44.887890 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.887814 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7"} err="failed to get container status \"d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7\": rpc error: code = NotFound desc = could not find container \"d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7\": container with ID starting with d3abdd34219acc40f67fb34473098971dee54544206e7c6b6d45ca2cd868f2d7 not found: ID does not exist" Sep 11 18:21:44.887890 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.887840 2579 scope.go:117] "RemoveContainer" containerID="291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc" Sep 11 18:21:44.888145 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:44.888123 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc\": container with ID starting with 291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc not found: ID does not exist" containerID="291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc" Sep 11 18:21:44.888186 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.888156 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc"} err="failed to get container status \"291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc\": rpc error: code = NotFound desc = could not find container \"291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc\": container with ID starting with 291bee811970eb03e4f45e072343b731eade4bdf64bfc1ca6a1b889426259fcc not found: ID does not exist" Sep 11 18:21:44.888483 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.888465 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj"] Sep 11 18:21:44.893308 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:44.893287 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-794d1c-openshift-default-64fcc6d786-8gndj"] Sep 11 18:21:46.445708 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:46.445674 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" path="/var/lib/kubelet/pods/bef14f41-9a89-4163-b099-9ed713c9ae3c/volumes" Sep 11 18:21:48.666009 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:48.665959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:49.177990 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:49.177961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:49.794711 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:49.794678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w"] Sep 11 18:21:49.795175 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:49.794932 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" podUID="c192bb39-b3dc-460c-9ff1-115899dcf9e7" containerName="maas-api" containerID="cri-o://802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1" gracePeriod=30 Sep 11 18:21:49.846654 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:49.846619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:49.980238 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:49.980145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:50.064400 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.064380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:50.075631 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.075610 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:21:50.122800 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.122772 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9b2w\" (UniqueName: \"kubernetes.io/projected/c192bb39-b3dc-460c-9ff1-115899dcf9e7-kube-api-access-f9b2w\") pod \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " Sep 11 18:21:50.122977 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.122806 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls\") pod \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " Sep 11 18:21:50.122977 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.122880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c192bb39-b3dc-460c-9ff1-115899dcf9e7-openshift-service-ca\") pod \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " Sep 11 18:21:50.122977 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.122984 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-metrics-certs\") pod \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\" (UID: \"c192bb39-b3dc-460c-9ff1-115899dcf9e7\") " Sep 11 18:21:50.123257 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.123226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c192bb39-b3dc-460c-9ff1-115899dcf9e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c192bb39-b3dc-460c-9ff1-115899dcf9e7" (UID: "c192bb39-b3dc-460c-9ff1-115899dcf9e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:21:50.123397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.123337 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c192bb39-b3dc-460c-9ff1-115899dcf9e7-openshift-service-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:50.125659 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.125622 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c192bb39-b3dc-460c-9ff1-115899dcf9e7" (UID: "c192bb39-b3dc-460c-9ff1-115899dcf9e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:21:50.125784 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.125738 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c192bb39-b3dc-460c-9ff1-115899dcf9e7" (UID: "c192bb39-b3dc-460c-9ff1-115899dcf9e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:21:50.125863 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.125805 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c192bb39-b3dc-460c-9ff1-115899dcf9e7-kube-api-access-f9b2w" (OuterVolumeSpecName: "kube-api-access-f9b2w") pod "c192bb39-b3dc-460c-9ff1-115899dcf9e7" (UID: "c192bb39-b3dc-460c-9ff1-115899dcf9e7"). InnerVolumeSpecName "kube-api-access-f9b2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:21:50.166436 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.166385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:50.224159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.224115 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:50.224159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.224149 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9b2w\" (UniqueName: \"kubernetes.io/projected/c192bb39-b3dc-460c-9ff1-115899dcf9e7-kube-api-access-f9b2w\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:50.224159 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.224165 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c192bb39-b3dc-460c-9ff1-115899dcf9e7-maas-api-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:21:50.263502 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.263467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:50.363949 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.363849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:50.598395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.598365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:50.900249 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.900212 2579 generic.go:356] "Generic (PLEG): container finished" podID="c192bb39-b3dc-460c-9ff1-115899dcf9e7" containerID="802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1" exitCode=0 Sep 11 18:21:50.900718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.900255 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" Sep 11 18:21:50.900718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.900292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" event={"ID":"c192bb39-b3dc-460c-9ff1-115899dcf9e7","Type":"ContainerDied","Data":"802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1"} Sep 11 18:21:50.900718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.900345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w" event={"ID":"c192bb39-b3dc-460c-9ff1-115899dcf9e7","Type":"ContainerDied","Data":"af2ccf5ea9cde1ddde4fdb1600a472e8c93c9a903831d8fe74ebfdb0363d1e19"} Sep 11 18:21:50.900718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.900369 2579 scope.go:117] "RemoveContainer" containerID="802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1" Sep 11 18:21:50.910341 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.910322 2579 scope.go:117] "RemoveContainer" containerID="802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1" Sep 11 18:21:50.910661 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:21:50.910643 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1\": container with ID starting with 802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1 not found: ID does not exist" containerID="802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1" Sep 11 18:21:50.910734 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.910675 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1"} err="failed to get container status \"802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1\": rpc error: code = NotFound desc = could not find container \"802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1\": container with ID starting with 802b63dee4fa3f36a9fc88487cd37b5121afd70492b19f1743882104d30fdbe1 not found: ID does not exist" Sep 11 18:21:50.919899 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.919836 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w"] Sep 11 18:21:50.921929 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.921907 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b05610-59bcd7fcf9-fvq2w"] Sep 11 18:21:50.983081 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:50.983051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:51.685484 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:51.685450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:52.445291 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:52.445247 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c192bb39-b3dc-460c-9ff1-115899dcf9e7" path="/var/lib/kubelet/pods/c192bb39-b3dc-460c-9ff1-115899dcf9e7/volumes" Sep 11 18:21:52.468156 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:52.468127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:53.152701 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:53.152664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:53.730768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:53.730738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:53.850200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:53.850166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:56.850463 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:56.850427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:56.948126 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:56.948097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:57.084846 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:57.084815 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf"] Sep 11 18:21:57.085082 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:57.085055 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" containerID="cri-o://743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570" gracePeriod=30 Sep 11 18:21:57.110665 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:57.110588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:58.128518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:58.128483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:58.872678 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:58.872644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:21:59.732289 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:21:59.732227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:00.416657 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:00.416616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:02.345303 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.345276 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:22:02.442834 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.442797 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-certs\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.442918 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-credential-socket\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.442962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-podinfo\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.442999 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-envoy\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443044 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:02.443161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443100 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-token\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443121 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-data\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443161 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkp9d\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-kube-api-access-kkp9d\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443371 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443166 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-socket\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443371 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443260 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:02.443499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443382 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:02.443499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istiod-ca-cert\") pod \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\" (UID: \"17f30f33-f227-43bf-b0cf-bc0b2bb2080f\") " Sep 11 18:22:02.443606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-data" (OuterVolumeSpecName: "istio-data") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:02.443893 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443863 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.443893 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443888 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.444090 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443904 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.444090 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.443918 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.444215 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.444174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:02.445686 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.445648 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:22:02.445808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.445718 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-kube-api-access-kkp9d" (OuterVolumeSpecName: "kube-api-access-kkp9d") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "kube-api-access-kkp9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:02.445808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.445737 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-token" (OuterVolumeSpecName: "istio-token") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:02.445808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.445777 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17f30f33-f227-43bf-b0cf-bc0b2bb2080f" (UID: "17f30f33-f227-43bf-b0cf-bc0b2bb2080f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:22:02.545356 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.545322 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.545356 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.545350 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.545356 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.545359 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.545619 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.545368 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkp9d\" (UniqueName: \"kubernetes.io/projected/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-kube-api-access-kkp9d\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.545619 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.545376 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f30f33-f227-43bf-b0cf-bc0b2bb2080f-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:02.967194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.967154 2579 generic.go:356] "Generic (PLEG): container finished" podID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerID="743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570" exitCode=0 Sep 11 18:22:02.967376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.967206 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" Sep 11 18:22:02.967376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.967226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" event={"ID":"17f30f33-f227-43bf-b0cf-bc0b2bb2080f","Type":"ContainerDied","Data":"743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570"} Sep 11 18:22:02.967376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.967280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf" event={"ID":"17f30f33-f227-43bf-b0cf-bc0b2bb2080f","Type":"ContainerDied","Data":"7aea2aa3ec6f716ba2de6edb3b4c2c998c963f4ac9ccd3e6806d998dc222c47c"} Sep 11 18:22:02.967376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.967310 2579 scope.go:117] "RemoveContainer" containerID="743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570" Sep 11 18:22:02.978596 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.978491 2579 scope.go:117] "RemoveContainer" containerID="5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6" Sep 11 18:22:02.987989 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.987901 2579 scope.go:117] "RemoveContainer" containerID="743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570" Sep 11 18:22:02.988325 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:02.988301 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570\": container with ID starting with 743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570 not found: ID does not exist" containerID="743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570" Sep 11 18:22:02.988407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.988338 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570"} err="failed to get container status \"743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570\": rpc error: code = NotFound desc = could not find container \"743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570\": container with ID starting with 743816ba99599af44f56297787c7ef734cdb13d2f2953c7b31715ef660e2c570 not found: ID does not exist" Sep 11 18:22:02.988407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.988363 2579 scope.go:117] "RemoveContainer" containerID="5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6" Sep 11 18:22:02.988719 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:02.988692 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6\": container with ID starting with 5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6 not found: ID does not exist" containerID="5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6" Sep 11 18:22:02.988818 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.988727 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6"} err="failed to get container status \"5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6\": rpc error: code = NotFound desc = could not find container \"5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6\": container with ID starting with 5d84f057a2a94e1203d7a3e021c34759386396530f0236ae6fc84b297ad8a6e6 not found: ID does not exist" Sep 11 18:22:02.990643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.990621 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf"] Sep 11 18:22:02.992490 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:02.992472 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b05610-openshift-default-646bbf8-frnmf"] Sep 11 18:22:04.446566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.446532 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" path="/var/lib/kubelet/pods/17f30f33-f227-43bf-b0cf-bc0b2bb2080f/volumes" Sep 11 18:22:04.476570 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.476537 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6"] Sep 11 18:22:04.476800 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.476774 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" podUID="65947d2f-2289-4d28-a0aa-bd95f04ad62b" containerName="maas-api" containerID="cri-o://4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab" gracePeriod=30 Sep 11 18:22:04.510793 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.510762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:04.619021 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.618991 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g"] Sep 11 18:22:04.619223 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.619197 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" podUID="a481d95a-e6ab-4098-a823-1da5a8450937" containerName="payload-processing" containerID="cri-o://fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927" gracePeriod=30 Sep 11 18:22:04.628623 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.628597 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs"] Sep 11 18:22:04.628845 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.628815 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" podUID="2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" containerName="payload-pre-processing" containerID="cri-o://789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0" gracePeriod=30 Sep 11 18:22:04.656191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.656167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:04.752997 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.752957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:04.756897 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.756854 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:22:04.826249 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.826215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:04.867772 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.867746 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcncx\" (UniqueName: \"kubernetes.io/projected/65947d2f-2289-4d28-a0aa-bd95f04ad62b-kube-api-access-qcncx\") pod \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " Sep 11 18:22:04.867959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.867802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-maas-api-tls\") pod \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " Sep 11 18:22:04.867959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.867853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-metrics-certs\") pod \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " Sep 11 18:22:04.867959 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.867880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65947d2f-2289-4d28-a0aa-bd95f04ad62b-openshift-service-ca\") pod \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\" (UID: \"65947d2f-2289-4d28-a0aa-bd95f04ad62b\") " Sep 11 18:22:04.868556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.868524 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65947d2f-2289-4d28-a0aa-bd95f04ad62b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65947d2f-2289-4d28-a0aa-bd95f04ad62b" (UID: "65947d2f-2289-4d28-a0aa-bd95f04ad62b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:04.871305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.871268 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65947d2f-2289-4d28-a0aa-bd95f04ad62b-kube-api-access-qcncx" (OuterVolumeSpecName: "kube-api-access-qcncx") pod "65947d2f-2289-4d28-a0aa-bd95f04ad62b" (UID: "65947d2f-2289-4d28-a0aa-bd95f04ad62b"). InnerVolumeSpecName "kube-api-access-qcncx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:04.872699 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.872672 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65947d2f-2289-4d28-a0aa-bd95f04ad62b" (UID: "65947d2f-2289-4d28-a0aa-bd95f04ad62b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:04.872699 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.872689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65947d2f-2289-4d28-a0aa-bd95f04ad62b" (UID: "65947d2f-2289-4d28-a0aa-bd95f04ad62b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:22:04.914575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.914545 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:22:04.923199 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.923167 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:22:04.968597 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.968562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-plugins-config-volume\") pod \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " Sep 11 18:22:04.968780 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.968620 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2949d\" (UniqueName: \"kubernetes.io/projected/a481d95a-e6ab-4098-a823-1da5a8450937-kube-api-access-2949d\") pod \"a481d95a-e6ab-4098-a823-1da5a8450937\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " Sep 11 18:22:04.968780 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.968685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8nwc\" (UniqueName: \"kubernetes.io/projected/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-kube-api-access-f8nwc\") pod \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\" (UID: \"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b\") " Sep 11 18:22:04.968780 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.968742 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a481d95a-e6ab-4098-a823-1da5a8450937-plugins-config-volume\") pod \"a481d95a-e6ab-4098-a823-1da5a8450937\" (UID: \"a481d95a-e6ab-4098-a823-1da5a8450937\") " Sep 11 18:22:04.969077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.969057 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcncx\" (UniqueName: \"kubernetes.io/projected/65947d2f-2289-4d28-a0aa-bd95f04ad62b-kube-api-access-qcncx\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:04.969172 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.969081 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-maas-api-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:04.969172 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.969095 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65947d2f-2289-4d28-a0aa-bd95f04ad62b-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:04.969172 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.969096 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" (UID: "2ce42301-6248-4ad9-a6dd-d04a0f2ed67b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:04.969172 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.969107 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65947d2f-2289-4d28-a0aa-bd95f04ad62b-openshift-service-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:04.969172 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.969139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a481d95a-e6ab-4098-a823-1da5a8450937-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a481d95a-e6ab-4098-a823-1da5a8450937" (UID: "a481d95a-e6ab-4098-a823-1da5a8450937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:04.970979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.970953 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a481d95a-e6ab-4098-a823-1da5a8450937-kube-api-access-2949d" (OuterVolumeSpecName: "kube-api-access-2949d") pod "a481d95a-e6ab-4098-a823-1da5a8450937" (UID: "a481d95a-e6ab-4098-a823-1da5a8450937"). InnerVolumeSpecName "kube-api-access-2949d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:04.971588 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.971564 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-kube-api-access-f8nwc" (OuterVolumeSpecName: "kube-api-access-f8nwc") pod "2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" (UID: "2ce42301-6248-4ad9-a6dd-d04a0f2ed67b"). InnerVolumeSpecName "kube-api-access-f8nwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:04.981969 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.981938 2579 generic.go:356] "Generic (PLEG): container finished" podID="2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" containerID="789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0" exitCode=0 Sep 11 18:22:04.981969 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.981977 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" Sep 11 18:22:04.982190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.982016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" event={"ID":"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b","Type":"ContainerDied","Data":"789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0"} Sep 11 18:22:04.982190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.982077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs" event={"ID":"2ce42301-6248-4ad9-a6dd-d04a0f2ed67b","Type":"ContainerDied","Data":"e4d5bb41c20727c2cbda62d13fcacbb3f3257b4cb6fd18e02747a734a87fe443"} Sep 11 18:22:04.982190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.982109 2579 scope.go:117] "RemoveContainer" containerID="789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0" Sep 11 18:22:04.983705 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.983635 2579 generic.go:356] "Generic (PLEG): container finished" podID="65947d2f-2289-4d28-a0aa-bd95f04ad62b" containerID="4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab" exitCode=0 Sep 11 18:22:04.983705 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.983684 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" Sep 11 18:22:04.983822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.983711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" event={"ID":"65947d2f-2289-4d28-a0aa-bd95f04ad62b","Type":"ContainerDied","Data":"4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab"} Sep 11 18:22:04.983822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.983786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6" event={"ID":"65947d2f-2289-4d28-a0aa-bd95f04ad62b","Type":"ContainerDied","Data":"be516cdd2be4e65771cc552442cee03b6278f0b2150b9f33410135caf0deb604"} Sep 11 18:22:04.985519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.985334 2579 generic.go:356] "Generic (PLEG): container finished" podID="a481d95a-e6ab-4098-a823-1da5a8450937" containerID="fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927" exitCode=0 Sep 11 18:22:04.985519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.985384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" event={"ID":"a481d95a-e6ab-4098-a823-1da5a8450937","Type":"ContainerDied","Data":"fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927"} Sep 11 18:22:04.985519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.985502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" event={"ID":"a481d95a-e6ab-4098-a823-1da5a8450937","Type":"ContainerDied","Data":"b31d077919dbad41f7bb22b0cc383c8e579e40ebc91057e8c02aa22883ba1db9"} Sep 11 18:22:04.985519 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.985392 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g" Sep 11 18:22:04.994191 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.994160 2579 scope.go:117] "RemoveContainer" containerID="789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0" Sep 11 18:22:04.994525 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:04.994505 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0\": container with ID starting with 789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0 not found: ID does not exist" containerID="789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0" Sep 11 18:22:04.994621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.994536 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0"} err="failed to get container status \"789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0\": rpc error: code = NotFound desc = could not find container \"789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0\": container with ID starting with 789b1bb1cc6825228366aba72944d44b64146a45cffcc546b14d7dab0f56abc0 not found: ID does not exist" Sep 11 18:22:04.994621 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:04.994557 2579 scope.go:117] "RemoveContainer" containerID="4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab" Sep 11 18:22:05.014886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.014856 2579 scope.go:117] "RemoveContainer" containerID="4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab" Sep 11 18:22:05.015026 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.015006 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs"] Sep 11 18:22:05.015236 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:05.015219 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab\": container with ID starting with 4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab not found: ID does not exist" containerID="4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab" Sep 11 18:22:05.015286 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.015242 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab"} err="failed to get container status \"4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab\": rpc error: code = NotFound desc = could not find container \"4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab\": container with ID starting with 4db5f3053a1cb322a4810a7fbbcbaf684777982bd81c5cbcafa9890dba84e0ab not found: ID does not exist" Sep 11 18:22:05.015286 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.015260 2579 scope.go:117] "RemoveContainer" containerID="fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927" Sep 11 18:22:05.019180 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.019149 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4e0b0b-57865b6dbb-dq2cs"] Sep 11 18:22:05.024376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.024355 2579 scope.go:117] "RemoveContainer" containerID="fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927" Sep 11 18:22:05.024711 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:05.024684 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927\": container with ID starting with fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927 not found: ID does not exist" containerID="fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927" Sep 11 18:22:05.024791 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.024723 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927"} err="failed to get container status \"fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927\": rpc error: code = NotFound desc = could not find container \"fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927\": container with ID starting with fdd69d0ee357ce698e480851ab6176a190c17f8068757154bcfdcfb5ca573927 not found: ID does not exist" Sep 11 18:22:05.033372 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.033347 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6"] Sep 11 18:22:05.036928 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.036899 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e0b0b-9897bd66c-6dkb6"] Sep 11 18:22:05.050071 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.050046 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g"] Sep 11 18:22:05.058083 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.058060 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4e0b0b-6f6c558bcc-5xl6g"] Sep 11 18:22:05.070074 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.070054 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:05.070074 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.070074 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2949d\" (UniqueName: \"kubernetes.io/projected/a481d95a-e6ab-4098-a823-1da5a8450937-kube-api-access-2949d\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:05.070190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.070084 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8nwc\" (UniqueName: \"kubernetes.io/projected/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b-kube-api-access-f8nwc\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:05.070190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.070094 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a481d95a-e6ab-4098-a823-1da5a8450937-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:05.170557 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.170522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:05.280729 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:05.280640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:06.165148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:06.165111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:06.445277 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:06.445192 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" path="/var/lib/kubelet/pods/2ce42301-6248-4ad9-a6dd-d04a0f2ed67b/volumes" Sep 11 18:22:06.445799 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:06.445777 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65947d2f-2289-4d28-a0aa-bd95f04ad62b" path="/var/lib/kubelet/pods/65947d2f-2289-4d28-a0aa-bd95f04ad62b/volumes" Sep 11 18:22:06.446162 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:06.446145 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a481d95a-e6ab-4098-a823-1da5a8450937" path="/var/lib/kubelet/pods/a481d95a-e6ab-4098-a823-1da5a8450937/volumes" Sep 11 18:22:06.887296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:06.887264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:07.740544 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:07.740511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:08.549162 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.549126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:08.557039 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.557010 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7"] Sep 11 18:22:08.557257 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.557234 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" podUID="4e9d9cd6-e6e0-4219-8347-bed285b66db9" containerName="payload-processing" containerID="cri-o://c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74" gracePeriod=30 Sep 11 18:22:08.563957 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.563926 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx"] Sep 11 18:22:08.564263 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.564190 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" podUID="e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" containerName="payload-pre-processing" containerID="cri-o://416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44" gracePeriod=30 Sep 11 18:22:08.648215 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.648184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:08.736537 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.736367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:08.814289 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.814135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:08.838032 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.838004 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:22:08.861389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.861358 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:22:08.916148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.916113 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67wng\" (UniqueName: \"kubernetes.io/projected/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-kube-api-access-67wng\") pod \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " Sep 11 18:22:08.916317 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.916164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt7p8\" (UniqueName: \"kubernetes.io/projected/4e9d9cd6-e6e0-4219-8347-bed285b66db9-kube-api-access-bt7p8\") pod \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " Sep 11 18:22:08.916317 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.916186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9d9cd6-e6e0-4219-8347-bed285b66db9-plugins-config-volume\") pod \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\" (UID: \"4e9d9cd6-e6e0-4219-8347-bed285b66db9\") " Sep 11 18:22:08.916317 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.916225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-plugins-config-volume\") pod \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\" (UID: \"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661\") " Sep 11 18:22:08.916612 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.916586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9d9cd6-e6e0-4219-8347-bed285b66db9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e9d9cd6-e6e0-4219-8347-bed285b66db9" (UID: "4e9d9cd6-e6e0-4219-8347-bed285b66db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:08.916698 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.916626 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" (UID: "e8acc2c4-c2d7-46fa-aeac-fbf6432cf661"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:22:08.918628 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.918603 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-kube-api-access-67wng" (OuterVolumeSpecName: "kube-api-access-67wng") pod "e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" (UID: "e8acc2c4-c2d7-46fa-aeac-fbf6432cf661"). InnerVolumeSpecName "kube-api-access-67wng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:08.918738 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:08.918716 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9d9cd6-e6e0-4219-8347-bed285b66db9-kube-api-access-bt7p8" (OuterVolumeSpecName: "kube-api-access-bt7p8") pod "4e9d9cd6-e6e0-4219-8347-bed285b66db9" (UID: "4e9d9cd6-e6e0-4219-8347-bed285b66db9"). InnerVolumeSpecName "kube-api-access-bt7p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:22:09.012787 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.012751 2579 generic.go:356] "Generic (PLEG): container finished" podID="e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" containerID="416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44" exitCode=0 Sep 11 18:22:09.012787 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.012791 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" Sep 11 18:22:09.012996 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.012837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" event={"ID":"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661","Type":"ContainerDied","Data":"416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44"} Sep 11 18:22:09.012996 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.012887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx" event={"ID":"e8acc2c4-c2d7-46fa-aeac-fbf6432cf661","Type":"ContainerDied","Data":"b54837dad220c19a854d06f127611e0645b10f1a7a92215d7eb86d6ba0cd1b1f"} Sep 11 18:22:09.012996 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.012911 2579 scope.go:117] "RemoveContainer" containerID="416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44" Sep 11 18:22:09.014680 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.014644 2579 generic.go:356] "Generic (PLEG): container finished" podID="4e9d9cd6-e6e0-4219-8347-bed285b66db9" containerID="c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74" exitCode=0 Sep 11 18:22:09.014802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.014686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" event={"ID":"4e9d9cd6-e6e0-4219-8347-bed285b66db9","Type":"ContainerDied","Data":"c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74"} Sep 11 18:22:09.014802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.014692 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" Sep 11 18:22:09.014802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.014728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7" event={"ID":"4e9d9cd6-e6e0-4219-8347-bed285b66db9","Type":"ContainerDied","Data":"d1c18453a46cf3dc9fc074fc7928125f9fc95c309ea9ca37f41c9e6ee91b7289"} Sep 11 18:22:09.017860 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.017834 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wng\" (UniqueName: \"kubernetes.io/projected/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-kube-api-access-67wng\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:09.017979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.017865 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt7p8\" (UniqueName: \"kubernetes.io/projected/4e9d9cd6-e6e0-4219-8347-bed285b66db9-kube-api-access-bt7p8\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:09.017979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.017881 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9d9cd6-e6e0-4219-8347-bed285b66db9-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:09.017979 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.017893 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:22:09.026397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.026308 2579 scope.go:117] "RemoveContainer" containerID="416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44" Sep 11 18:22:09.026866 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:09.026837 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44\": container with ID starting with 416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44 not found: ID does not exist" containerID="416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44" Sep 11 18:22:09.026942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.026873 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44"} err="failed to get container status \"416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44\": rpc error: code = NotFound desc = could not find container \"416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44\": container with ID starting with 416b5995e89872793cc1bb00d201009a8a41884d3ad24177b07f65062fd45f44 not found: ID does not exist" Sep 11 18:22:09.026942 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.026892 2579 scope.go:117] "RemoveContainer" containerID="c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74" Sep 11 18:22:09.037307 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.037287 2579 scope.go:117] "RemoveContainer" containerID="c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74" Sep 11 18:22:09.037687 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:22:09.037666 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74\": container with ID starting with c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74 not found: ID does not exist" containerID="c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74" Sep 11 18:22:09.037793 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.037693 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74"} err="failed to get container status \"c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74\": rpc error: code = NotFound desc = could not find container \"c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74\": container with ID starting with c1c8fbad33d30343162675ef24ee57e35e6b9ebae8f2cd9fa5ddd28e3b571e74 not found: ID does not exist" Sep 11 18:22:09.040478 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.040407 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx"] Sep 11 18:22:09.043127 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.043104 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-90057a-75bc68ff8-bpvwx"] Sep 11 18:22:09.054850 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.054822 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7"] Sep 11 18:22:09.058777 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.058748 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-90057a-57949c658c-5ktm7"] Sep 11 18:22:09.104278 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.104198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:09.232706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.232669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:09.954886 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:09.954849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:10.443943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:10.443912 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9d9cd6-e6e0-4219-8347-bed285b66db9" path="/var/lib/kubelet/pods/4e9d9cd6-e6e0-4219-8347-bed285b66db9/volumes" Sep 11 18:22:10.444310 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:10.444295 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" path="/var/lib/kubelet/pods/e8acc2c4-c2d7-46fa-aeac-fbf6432cf661/volumes" Sep 11 18:22:10.676440 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:10.676379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:11.835497 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:11.835464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:11.935670 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:11.935643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:12.093363 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:12.093283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:15.324549 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:15.324518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:15.424151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:15.424117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:15.576601 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:15.576509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:23.280193 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:23.280158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:23.381040 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:23.381005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:29.930402 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:29.930368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:30.464482 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:30.464401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:30.951152 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:30.951117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:31.218307 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:31.218220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:31.797046 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:31.797010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:38.138858 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:38.138781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:38.296246 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:38.296207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:38.414681 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:38.414591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:38.525560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:38.525528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:38.937259 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:38.937226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:22:44.551697 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:22:44.551660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:03.238584 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:23:03.238545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:23:18.251531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:23:18.251485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:03.652642 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:24:03.652605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:04.704393 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:24:04.704358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:22.366834 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:24:22.366796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:43.042624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:24:43.042590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:24:58.749016 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:24:58.748951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:19.744593 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:19.744565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:33.820874 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:33.820842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:34.846657 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:34.846615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.269556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.269515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.535863 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.535772 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj"] Sep 11 18:25:35.536329 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536308 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" Sep 11 18:25:35.536395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536333 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" Sep 11 18:25:35.536395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536353 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9d9cd6-e6e0-4219-8347-bed285b66db9" containerName="payload-processing" Sep 11 18:25:35.536395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536362 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9d9cd6-e6e0-4219-8347-bed285b66db9" containerName="payload-processing" Sep 11 18:25:35.536395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536373 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783cd5d7-fbda-4148-a98a-e593a63341c0" containerName="payload-pre-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536434 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="783cd5d7-fbda-4148-a98a-e593a63341c0" containerName="payload-pre-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536448 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536459 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536472 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65947d2f-2289-4d28-a0aa-bd95f04ad62b" containerName="maas-api" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536481 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="65947d2f-2289-4d28-a0aa-bd95f04ad62b" containerName="maas-api" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536512 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536522 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536535 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a9b339-ff1f-4411-a993-b9675d71e284" containerName="payload-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536544 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a9b339-ff1f-4411-a993-b9675d71e284" containerName="payload-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536579 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="700f316d-1ed1-4d5e-acd4-2e938f2a47a0" containerName="payload-pre-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536587 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="700f316d-1ed1-4d5e-acd4-2e938f2a47a0" containerName="payload-pre-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536600 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536609 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536617 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" containerName="payload-pre-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536624 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" containerName="payload-pre-processing" Sep 11 18:25:35.536624 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536633 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c192bb39-b3dc-460c-9ff1-115899dcf9e7" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536641 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c192bb39-b3dc-460c-9ff1-115899dcf9e7" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536660 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4614eb16-3474-45ce-a151-51bc8c239020" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536668 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4614eb16-3474-45ce-a151-51bc8c239020" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536676 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536683 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536693 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a481d95a-e6ab-4098-a823-1da5a8450937" containerName="payload-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536700 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a481d95a-e6ab-4098-a823-1da5a8450937" containerName="payload-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536708 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" containerName="payload-pre-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536716 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" containerName="payload-pre-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536811 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9d9cd6-e6e0-4219-8347-bed285b66db9" containerName="payload-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536824 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536831 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c192bb39-b3dc-460c-9ff1-115899dcf9e7" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536862 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="783cd5d7-fbda-4148-a98a-e593a63341c0" containerName="payload-pre-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536873 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="65873c38-8f3d-4261-972e-23c9b1d91664" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536880 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="700f316d-1ed1-4d5e-acd4-2e938f2a47a0" containerName="payload-pre-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536888 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536896 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4614eb16-3474-45ce-a151-51bc8c239020" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536904 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce42301-6248-4ad9-a6dd-d04a0f2ed67b" containerName="payload-pre-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536911 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a481d95a-e6ab-4098-a823-1da5a8450937" containerName="payload-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536919 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8acc2c4-c2d7-46fa-aeac-fbf6432cf661" containerName="payload-pre-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536928 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536936 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="65947d2f-2289-4d28-a0aa-bd95f04ad62b" containerName="maas-api" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536945 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536952 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a9b339-ff1f-4411-a993-b9675d71e284" containerName="payload-processing" Sep 11 18:25:35.537499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.536960 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef14f41-9a89-4163-b099-9ed713c9ae3c" containerName="istio-proxy" Sep 11 18:25:35.540302 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.540284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.543258 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.543237 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-be394b-openshift-default-dockercfg-fz82m\"" Sep 11 18:25:35.550747 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.550717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj"] Sep 11 18:25:35.568573 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.568517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.698806 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.698767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istiod-ca-cert\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.698806 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.698811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl5qp\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-kube-api-access-kl5qp\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.698842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-podinfo\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.698903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-envoy\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.698956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-socket\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.698981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-data\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.699003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-certs\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.699041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-token\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.699200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.699071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-credential-socket\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.799676 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.799587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-credential-socket\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.799676 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.799652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istiod-ca-cert\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.799908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.799682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl5qp\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-kube-api-access-kl5qp\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.799908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.799715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-podinfo\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.799908 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.799745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-envoy\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800149 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.799953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-socket\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800149 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-data\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800149 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-certs\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800149 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-credential-socket\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800149 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-token\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800461 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-socket\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800461 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-certs\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-data\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.800706 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.800687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istiod-ca-cert\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.802213 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.802194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-envoy\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.802266 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.802255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-podinfo\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.808121 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.808091 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-token\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.808248 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.808232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl5qp\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-kube-api-access-kl5qp\") pod \"e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:35.824517 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.824472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:35.854146 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:35.854083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:36.127612 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:36.127589 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj"] Sep 11 18:25:36.129349 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:25:36.129320 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3f7042_cc4c_46e2_a7c3_a1e251205907.slice/crio-9637ee5baeff2818b04ab1052937d78b5236845ae226679d8d122d47b8fa46dc WatchSource:0}: Error finding container 9637ee5baeff2818b04ab1052937d78b5236845ae226679d8d122d47b8fa46dc: Status 404 returned error can't find the container with id 9637ee5baeff2818b04ab1052937d78b5236845ae226679d8d122d47b8fa46dc Sep 11 18:25:36.131701 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:36.131667 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:25:36.131783 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:36.131751 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:25:36.131825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:36.131789 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:25:36.978377 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:36.978335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" event={"ID":"8c3f7042-cc4c-46e2-a7c3-a1e251205907","Type":"ContainerStarted","Data":"0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466"} Sep 11 18:25:36.978833 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:36.978388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" event={"ID":"8c3f7042-cc4c-46e2-a7c3-a1e251205907","Type":"ContainerStarted","Data":"9637ee5baeff2818b04ab1052937d78b5236845ae226679d8d122d47b8fa46dc"} Sep 11 18:25:37.004677 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:37.004620 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podStartSLOduration=2.004601096 podStartE2EDuration="2.004601096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:37.001795753 +0000 UTC m=+1881.032857006" watchObservedRunningTime="2026-09-11 18:25:37.004601096 +0000 UTC m=+1881.035662326" Sep 11 18:25:37.854927 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:37.854842 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:37.860368 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:37.860339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:37.983244 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:37.983208 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:37.984294 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:37.984271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:38.267742 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:38.267701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:38.912312 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:38.912275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:39.012073 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:39.012034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:39.863478 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:39.863445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:40.734387 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.734360 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p"] Sep 11 18:25:40.734760 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.734746 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" Sep 11 18:25:40.734809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.734762 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f30f33-f227-43bf-b0cf-bc0b2bb2080f" containerName="istio-proxy" Sep 11 18:25:40.738026 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.738010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.740953 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.740928 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-be394b-serving-cert\"" Sep 11 18:25:40.741086 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.741046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2sf5b\"" Sep 11 18:25:40.747061 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.747036 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p"] Sep 11 18:25:40.748967 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.748941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklhb\" (UniqueName: \"kubernetes.io/projected/6488462f-742d-42d5-924f-05251050199b-kube-api-access-mklhb\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.749200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.749176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-metrics-certs\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.749303 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.749284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6488462f-742d-42d5-924f-05251050199b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.749353 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.749325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.766316 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.766249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:40.839800 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.839734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:40.849987 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.849959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6488462f-742d-42d5-924f-05251050199b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.850066 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.849996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.850066 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.850036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklhb\" (UniqueName: \"kubernetes.io/projected/6488462f-742d-42d5-924f-05251050199b-kube-api-access-mklhb\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.850151 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.850077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-metrics-certs\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.850705 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.850683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6488462f-742d-42d5-924f-05251050199b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.852627 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.852603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.852718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.852685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-metrics-certs\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.859406 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.859381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklhb\" (UniqueName: \"kubernetes.io/projected/6488462f-742d-42d5-924f-05251050199b-kube-api-access-mklhb\") pod \"maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:40.920183 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:40.920145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:41.049991 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.049895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:41.259906 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.259869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:41.322389 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.322365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p"] Sep 11 18:25:41.324750 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:25:41.324719 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6488462f_742d_42d5_924f_05251050199b.slice/crio-298feb3009852c7a814ebff54e361d8f2ffada2e54f28349d13c3487a9a7f2f2 WatchSource:0}: Error finding container 298feb3009852c7a814ebff54e361d8f2ffada2e54f28349d13c3487a9a7f2f2: Status 404 returned error can't find the container with id 298feb3009852c7a814ebff54e361d8f2ffada2e54f28349d13c3487a9a7f2f2 Sep 11 18:25:41.326600 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.326581 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:25:41.391311 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.391276 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw"] Sep 11 18:25:41.395835 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.395818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.399148 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.399126 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-be394b-dockercfg-x5hhm\"" Sep 11 18:25:41.399290 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.399239 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be394b\"" Sep 11 18:25:41.405209 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.405185 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw"] Sep 11 18:25:41.405539 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.405519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:25:41.455480 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.455448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77z59\" (UniqueName: \"kubernetes.io/projected/c8ebddde-95e3-4462-aa05-86a3532aedf6-kube-api-access-77z59\") pod \"payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.455781 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.455705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ebddde-95e3-4462-aa05-86a3532aedf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.556816 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.556773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77z59\" (UniqueName: \"kubernetes.io/projected/c8ebddde-95e3-4462-aa05-86a3532aedf6-kube-api-access-77z59\") pod \"payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.556995 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.556881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ebddde-95e3-4462-aa05-86a3532aedf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.557537 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.557520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ebddde-95e3-4462-aa05-86a3532aedf6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.565737 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.565715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77z59\" (UniqueName: \"kubernetes.io/projected/c8ebddde-95e3-4462-aa05-86a3532aedf6-kube-api-access-77z59\") pod \"payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.708165 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.708133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:41.981259 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:41.981231 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw"] Sep 11 18:25:41.983635 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:25:41.983600 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ebddde_95e3_4462_aa05_86a3532aedf6.slice/crio-6c00c0f10e4875c647557cc00b855ab341eec1ed0c330f993cee25d9c3de00ba WatchSource:0}: Error finding container 6c00c0f10e4875c647557cc00b855ab341eec1ed0c330f993cee25d9c3de00ba: Status 404 returned error can't find the container with id 6c00c0f10e4875c647557cc00b855ab341eec1ed0c330f993cee25d9c3de00ba Sep 11 18:25:42.005894 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:42.005856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" event={"ID":"c8ebddde-95e3-4462-aa05-86a3532aedf6","Type":"ContainerStarted","Data":"6c00c0f10e4875c647557cc00b855ab341eec1ed0c330f993cee25d9c3de00ba"} Sep 11 18:25:42.007785 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:42.007751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" event={"ID":"6488462f-742d-42d5-924f-05251050199b","Type":"ContainerStarted","Data":"41b47d8ed3b69323b9643d31e31b1915c735d627b1567b0361e9ef253ac0b65f"} Sep 11 18:25:42.007891 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:42.007809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" event={"ID":"6488462f-742d-42d5-924f-05251050199b","Type":"ContainerStarted","Data":"298feb3009852c7a814ebff54e361d8f2ffada2e54f28349d13c3487a9a7f2f2"} Sep 11 18:25:42.007891 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:42.007871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:42.028038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:42.027978 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" podStartSLOduration=1.6189215689999998 podStartE2EDuration="2.027954357s" totalImagesPullingTime="409.032788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:42.025993777 +0000 UTC m=+1886.057055009" watchObservedRunningTime="2026-09-11 18:25:42.027954357 +0000 UTC m=+1886.059015587" Sep 11 18:25:43.014504 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:43.014463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" event={"ID":"c8ebddde-95e3-4462-aa05-86a3532aedf6","Type":"ContainerStarted","Data":"6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421"} Sep 11 18:25:43.014998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:43.014732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:43.033782 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:43.033736 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" podStartSLOduration=2.033719646 podStartE2EDuration="2.033719646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:43.030887011 +0000 UTC m=+1887.061948268" watchObservedRunningTime="2026-09-11 18:25:43.033719646 +0000 UTC m=+1887.064780912" Sep 11 18:25:48.021643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:48.021607 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:25:54.022627 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:54.022592 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:25:54.025534 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:54.025507 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 18:25:54.025671 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:54.025567 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 18:25:55.082293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:55.082177 2579 generic.go:356] "Generic (PLEG): container finished" podID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerID="0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466" exitCode=0 Sep 11 18:25:55.082797 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:55.082330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" event={"ID":"8c3f7042-cc4c-46e2-a7c3-a1e251205907","Type":"ContainerDied","Data":"0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466"} Sep 11 18:25:55.083040 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:55.083012 2579 scope.go:117] "RemoveContainer" containerID="0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466" Sep 11 18:25:55.088218 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:55.088188 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:25:55.088311 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:55.088261 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:25:55.088311 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:55.088293 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:25:56.089689 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:56.089647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" event={"ID":"8c3f7042-cc4c-46e2-a7c3-a1e251205907","Type":"ContainerStarted","Data":"141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315"} Sep 11 18:25:56.854965 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:56.854918 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:25:56.855165 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:56.855116 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 18:25:56.855228 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:56.855191 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 18:25:57.855407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:57.855373 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 18:25:57.855802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:57.855440 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 18:25:58.855440 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:58.855381 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 18:25:58.855821 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:58.855449 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 18:25:59.854787 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:59.854752 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 18:25:59.854964 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:25:59.854844 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 18:26:00.854845 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:00.854807 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 11 18:26:00.855241 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:00.854861 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 11 18:26:01.858962 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:01.858928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:26:01.859384 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:01.859182 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:26:01.860013 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:01.859995 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:26:03.903324 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:03.903293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.003331 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:04.003295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.273358 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:04.273270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.578274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:04.578182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.772932 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:04.772898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:04.882395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:04.882361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:05.045709 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:05.045673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:05.589572 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:05.589541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:06.415219 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:06.415184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:07.677056 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:07.677018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:08.392980 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:08.392931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:08.493037 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:08.492985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:09.276924 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:09.276888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:09.965827 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:09.965793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:10.676651 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:10.676592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:10.818867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:10.818820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:10.966359 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:10.966278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:11.168656 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:11.168618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:11.264229 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:11.264148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:13.238047 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:13.238009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:13.339128 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:13.339087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:32.182809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:32.182774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:33.063556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:33.063503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:33.414750 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:33.414709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:33.854113 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:33.854024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:34.657469 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:34.657438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:35.336900 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:35.336869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:36.008052 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:36.008018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:26:44.922152 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:26:44.922120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:07.648003 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:07.647917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:09.126664 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.126633 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw_c8ebddde-95e3-4462-aa05-86a3532aedf6/payload-processing/0.log" Sep 11 18:27:09.662177 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.662146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:09.934264 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.934169 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k"] Sep 11 18:27:09.937060 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.937040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.939811 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.939791 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-b2aace-openshift-default-dockercfg-fdkct\"" Sep 11 18:27:09.954025 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.953997 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k"] Sep 11 18:27:09.966504 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.966469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:09.999086 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbhs\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-kube-api-access-mlbhs\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999086 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-envoy\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999313 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-data\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999313 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac59e7ad-65d1-454c-8130-b2b0443ea379-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999394 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-certs\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999394 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-socket\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999500 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-podinfo\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999500 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-credential-socket\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:09.999579 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:09.999507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-token\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.066875 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.066845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:10.100118 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbhs\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-kube-api-access-mlbhs\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-envoy\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-data\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac59e7ad-65d1-454c-8130-b2b0443ea379-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100296 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-certs\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-socket\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-podinfo\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-credential-socket\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100535 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-token\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100808 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-socket\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100945 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-certs\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.100945 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-credential-socket\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.101031 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.100957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-data\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.101264 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.101241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac59e7ad-65d1-454c-8130-b2b0443ea379-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.102692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.102673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-envoy\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.102927 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.102910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-podinfo\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.108639 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.108620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-token\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.108763 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.108627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbhs\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-kube-api-access-mlbhs\") pod \"e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.249239 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.249150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:10.525577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.525548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k"] Sep 11 18:27:10.527228 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:27:10.527200 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac59e7ad_65d1_454c_8130_b2b0443ea379.slice/crio-5e7b04816e76d069372dbc3d1e1b03cbb28da5870f523d888801a50774259ba5 WatchSource:0}: Error finding container 5e7b04816e76d069372dbc3d1e1b03cbb28da5870f523d888801a50774259ba5: Status 404 returned error can't find the container with id 5e7b04816e76d069372dbc3d1e1b03cbb28da5870f523d888801a50774259ba5 Sep 11 18:27:10.529541 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.529509 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:27:10.529655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.529586 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:27:10.529655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:10.529640 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:27:11.450271 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:11.450233 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" event={"ID":"ac59e7ad-65d1-454c-8130-b2b0443ea379","Type":"ContainerStarted","Data":"f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb"} Sep 11 18:27:11.450747 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:11.450285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" event={"ID":"ac59e7ad-65d1-454c-8130-b2b0443ea379","Type":"ContainerStarted","Data":"5e7b04816e76d069372dbc3d1e1b03cbb28da5870f523d888801a50774259ba5"} Sep 11 18:27:11.474552 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:11.474506 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podStartSLOduration=2.474491575 podStartE2EDuration="2.474491575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:11.472121834 +0000 UTC m=+1975.503183103" watchObservedRunningTime="2026-09-11 18:27:11.474491575 +0000 UTC m=+1975.505552804" Sep 11 18:27:12.249873 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:12.249836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:12.255575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:12.255545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:12.454964 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:12.454930 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:12.456021 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:12.456003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:12.662293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:12.662264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:13.447556 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:13.447519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:13.548019 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:13.547985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:14.269754 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:14.269721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:15.153155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:15.153116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:15.851833 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:15.851801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:16.608436 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:16.608398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:16.710464 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:16.710431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:16.919488 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:16.919459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:17.109786 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:17.109753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:17.205933 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:17.205843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:33.562645 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:33.562553 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerID="f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb" exitCode=0 Sep 11 18:27:33.563029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:33.562636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" event={"ID":"ac59e7ad-65d1-454c-8130-b2b0443ea379","Type":"ContainerDied","Data":"f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb"} Sep 11 18:27:33.563029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:33.563013 2579 scope.go:117] "RemoveContainer" containerID="f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb" Sep 11 18:27:33.564260 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:33.564233 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:27:33.564318 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:33.564299 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:27:33.564358 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:33.564330 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 11 18:27:34.574527 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:34.574373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" event={"ID":"ac59e7ad-65d1-454c-8130-b2b0443ea379","Type":"ContainerStarted","Data":"836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef"} Sep 11 18:27:35.249650 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:35.249610 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:35.249949 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:35.249929 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:35.249997 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:35.249975 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 18:27:36.250030 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:36.249995 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:36.250412 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:36.250067 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 18:27:37.249559 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:37.249522 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:37.249774 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:37.249582 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 18:27:38.250618 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:38.250575 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:38.251034 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:38.250650 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 18:27:39.249562 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:39.249528 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 18:27:39.249822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:39.249611 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 18:27:39.952284 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:39.952236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:40.254463 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:40.254364 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:40.254643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:40.254620 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:40.255407 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:40.255386 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:27:40.647169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:40.647130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:41.368165 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:41.368120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:42.077862 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:42.077818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:42.755213 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:42.755176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:43.516079 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:43.516040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:43.677394 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:43.677361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:44.644550 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:44.644501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:45.352166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:45.352124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:46.268655 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:46.268621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:47.088373 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:47.088343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:47.477119 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:47.477086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:47.577413 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:47.577375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:47.930469 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:47.930437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:48.664000 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:48.663968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:55.435539 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:55.435500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:55.521249 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:55.521216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:55.588900 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:55.588871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:55.656462 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:55.656413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:56.026531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:56.026494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:56.129109 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:56.129073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:56.735807 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:56.735774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:57.427692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:57.427652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:58.155178 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:58.155125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:27:58.955452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:27:58.955389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:02.241558 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:02.241522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:02.347325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:02.347283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:02.478306 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:02.478271 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k"] Sep 11 18:28:02.478578 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:02.478537 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" containerID="cri-o://836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef" gracePeriod=30 Sep 11 18:28:02.502896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:02.502811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:06.385178 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:06.385143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:07.196403 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.196370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:07.736195 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.736169 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:28:07.749029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.748956 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerID="836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef" exitCode=0 Sep 11 18:28:07.749029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.749008 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" Sep 11 18:28:07.749029 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.749018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" event={"ID":"ac59e7ad-65d1-454c-8130-b2b0443ea379","Type":"ContainerDied","Data":"836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef"} Sep 11 18:28:07.749248 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.749065 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k" event={"ID":"ac59e7ad-65d1-454c-8130-b2b0443ea379","Type":"ContainerDied","Data":"5e7b04816e76d069372dbc3d1e1b03cbb28da5870f523d888801a50774259ba5"} Sep 11 18:28:07.749248 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.749088 2579 scope.go:117] "RemoveContainer" containerID="836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef" Sep 11 18:28:07.770348 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.770319 2579 scope.go:117] "RemoveContainer" containerID="f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb" Sep 11 18:28:07.780966 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.780942 2579 scope.go:117] "RemoveContainer" containerID="836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef" Sep 11 18:28:07.781280 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:28:07.781260 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef\": container with ID starting with 836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef not found: ID does not exist" containerID="836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef" Sep 11 18:28:07.781342 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.781289 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef"} err="failed to get container status \"836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef\": rpc error: code = NotFound desc = could not find container \"836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef\": container with ID starting with 836dde6452249f3ecac6de691940661c3995c716eb39c4aff2f4a41391a920ef not found: ID does not exist" Sep 11 18:28:07.781342 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.781307 2579 scope.go:117] "RemoveContainer" containerID="f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb" Sep 11 18:28:07.781641 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:28:07.781617 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb\": container with ID starting with f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb not found: ID does not exist" containerID="f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb" Sep 11 18:28:07.781710 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.781644 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb"} err="failed to get container status \"f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb\": rpc error: code = NotFound desc = could not find container \"f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb\": container with ID starting with f943d4f47271098359ef449236baba13372b282764de7d0452a79987aa273afb not found: ID does not exist" Sep 11 18:28:07.797980 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.797953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-envoy\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798130 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.797991 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-podinfo\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798130 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798018 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-credential-socket\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798130 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798051 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac59e7ad-65d1-454c-8130-b2b0443ea379-istiod-ca-cert\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798130 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-data\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798155 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-certs\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798180 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-socket\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798257 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlbhs\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-kube-api-access-mlbhs\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798376 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-token\") pod \"ac59e7ad-65d1-454c-8130-b2b0443ea379\" (UID: \"ac59e7ad-65d1-454c-8130-b2b0443ea379\") " Sep 11 18:28:07.798638 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798534 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:07.798702 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798596 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac59e7ad-65d1-454c-8130-b2b0443ea379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:28:07.798702 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798668 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:07.798804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798782 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.798804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798800 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.798907 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798815 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac59e7ad-65d1-454c-8130-b2b0443ea379-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.798907 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.798887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:07.799155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.799132 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:07.800612 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.800586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:28:07.800723 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.800626 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:28:07.800798 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.800751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:28:07.801036 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.801021 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-kube-api-access-mlbhs" (OuterVolumeSpecName: "kube-api-access-mlbhs") pod "ac59e7ad-65d1-454c-8130-b2b0443ea379" (UID: "ac59e7ad-65d1-454c-8130-b2b0443ea379"). InnerVolumeSpecName "kube-api-access-mlbhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:28:07.884643 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.884604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:07.899324 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.899288 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.899324 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.899316 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.899324 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.899325 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbhs\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-kube-api-access-mlbhs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.899324 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.899333 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.899713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.899342 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:07.899713 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:07.899348 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac59e7ad-65d1-454c-8130-b2b0443ea379-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:08.078859 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:08.078824 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k"] Sep 11 18:28:08.084299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:08.084271 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b2aace-openshift-default-676d5466fb-8fd2k"] Sep 11 18:28:08.443853 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:08.443817 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" path="/var/lib/kubelet/pods/ac59e7ad-65d1-454c-8130-b2b0443ea379/volumes" Sep 11 18:28:08.626576 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:08.626542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:09.355775 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:09.355747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:10.119838 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:10.119797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:21.685208 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.685166 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p"] Sep 11 18:28:21.685830 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.685471 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" podUID="6488462f-742d-42d5-924f-05251050199b" containerName="maas-api" containerID="cri-o://41b47d8ed3b69323b9643d31e31b1915c735d627b1567b0361e9ef253ac0b65f" gracePeriod=30 Sep 11 18:28:21.720970 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.720932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:21.742814 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.742779 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw"] Sep 11 18:28:21.743334 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.743275 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" podUID="c8ebddde-95e3-4462-aa05-86a3532aedf6" containerName="payload-processing" containerID="cri-o://6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421" gracePeriod=30 Sep 11 18:28:21.751915 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:28:21.751756 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6488462f_742d_42d5_924f_05251050199b.slice/crio-conmon-41b47d8ed3b69323b9643d31e31b1915c735d627b1567b0361e9ef253ac0b65f.scope\": RecentStats: unable to find data in memory cache]" Sep 11 18:28:21.819951 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.819898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:21.826501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.826471 2579 generic.go:356] "Generic (PLEG): container finished" podID="6488462f-742d-42d5-924f-05251050199b" containerID="41b47d8ed3b69323b9643d31e31b1915c735d627b1567b0361e9ef253ac0b65f" exitCode=0 Sep 11 18:28:21.826646 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.826548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" event={"ID":"6488462f-742d-42d5-924f-05251050199b","Type":"ContainerDied","Data":"41b47d8ed3b69323b9643d31e31b1915c735d627b1567b0361e9ef253ac0b65f"} Sep 11 18:28:21.917798 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.916765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:21.988698 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:21.988568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:22.023292 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.023266 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:28:22.026856 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.026825 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:28:22.146838 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.146807 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77z59\" (UniqueName: \"kubernetes.io/projected/c8ebddde-95e3-4462-aa05-86a3532aedf6-kube-api-access-77z59\") pod \"c8ebddde-95e3-4462-aa05-86a3532aedf6\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " Sep 11 18:28:22.146838 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.146845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-maas-api-tls\") pod \"6488462f-742d-42d5-924f-05251050199b\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " Sep 11 18:28:22.147092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.146876 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mklhb\" (UniqueName: \"kubernetes.io/projected/6488462f-742d-42d5-924f-05251050199b-kube-api-access-mklhb\") pod \"6488462f-742d-42d5-924f-05251050199b\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " Sep 11 18:28:22.147092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.146914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-metrics-certs\") pod \"6488462f-742d-42d5-924f-05251050199b\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " Sep 11 18:28:22.147092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.146949 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6488462f-742d-42d5-924f-05251050199b-openshift-service-ca\") pod \"6488462f-742d-42d5-924f-05251050199b\" (UID: \"6488462f-742d-42d5-924f-05251050199b\") " Sep 11 18:28:22.147092 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.147001 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ebddde-95e3-4462-aa05-86a3532aedf6-plugins-config-volume\") pod \"c8ebddde-95e3-4462-aa05-86a3532aedf6\" (UID: \"c8ebddde-95e3-4462-aa05-86a3532aedf6\") " Sep 11 18:28:22.147345 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.147306 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6488462f-742d-42d5-924f-05251050199b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6488462f-742d-42d5-924f-05251050199b" (UID: "6488462f-742d-42d5-924f-05251050199b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:28:22.147867 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.147667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ebddde-95e3-4462-aa05-86a3532aedf6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8ebddde-95e3-4462-aa05-86a3532aedf6" (UID: "c8ebddde-95e3-4462-aa05-86a3532aedf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:28:22.149330 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.149275 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6488462f-742d-42d5-924f-05251050199b" (UID: "6488462f-742d-42d5-924f-05251050199b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:28:22.149452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.149336 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6488462f-742d-42d5-924f-05251050199b" (UID: "6488462f-742d-42d5-924f-05251050199b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:28:22.149452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.149350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6488462f-742d-42d5-924f-05251050199b-kube-api-access-mklhb" (OuterVolumeSpecName: "kube-api-access-mklhb") pod "6488462f-742d-42d5-924f-05251050199b" (UID: "6488462f-742d-42d5-924f-05251050199b"). InnerVolumeSpecName "kube-api-access-mklhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:28:22.149452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.149379 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ebddde-95e3-4462-aa05-86a3532aedf6-kube-api-access-77z59" (OuterVolumeSpecName: "kube-api-access-77z59") pod "c8ebddde-95e3-4462-aa05-86a3532aedf6" (UID: "c8ebddde-95e3-4462-aa05-86a3532aedf6"). InnerVolumeSpecName "kube-api-access-77z59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:28:22.248163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.248082 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77z59\" (UniqueName: \"kubernetes.io/projected/c8ebddde-95e3-4462-aa05-86a3532aedf6-kube-api-access-77z59\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.248163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.248113 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-maas-api-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.248163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.248122 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklhb\" (UniqueName: \"kubernetes.io/projected/6488462f-742d-42d5-924f-05251050199b-kube-api-access-mklhb\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.248163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.248130 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6488462f-742d-42d5-924f-05251050199b-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.248163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.248138 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6488462f-742d-42d5-924f-05251050199b-openshift-service-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.248163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.248146 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ebddde-95e3-4462-aa05-86a3532aedf6-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:28:22.291560 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.291529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:22.389955 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.389920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:22.832597 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.832569 2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ebddde-95e3-4462-aa05-86a3532aedf6" containerID="6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421" exitCode=0 Sep 11 18:28:22.833051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.832608 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" Sep 11 18:28:22.833051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.832657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" event={"ID":"c8ebddde-95e3-4462-aa05-86a3532aedf6","Type":"ContainerDied","Data":"6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421"} Sep 11 18:28:22.833051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.832709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw" event={"ID":"c8ebddde-95e3-4462-aa05-86a3532aedf6","Type":"ContainerDied","Data":"6c00c0f10e4875c647557cc00b855ab341eec1ed0c330f993cee25d9c3de00ba"} Sep 11 18:28:22.833051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.832730 2579 scope.go:117] "RemoveContainer" containerID="6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421" Sep 11 18:28:22.834112 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.834089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" event={"ID":"6488462f-742d-42d5-924f-05251050199b","Type":"ContainerDied","Data":"298feb3009852c7a814ebff54e361d8f2ffada2e54f28349d13c3487a9a7f2f2"} Sep 11 18:28:22.834112 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.834107 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p" Sep 11 18:28:22.841453 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.841431 2579 scope.go:117] "RemoveContainer" containerID="6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421" Sep 11 18:28:22.841799 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:28:22.841779 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421\": container with ID starting with 6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421 not found: ID does not exist" containerID="6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421" Sep 11 18:28:22.841849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.841809 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421"} err="failed to get container status \"6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421\": rpc error: code = NotFound desc = could not find container \"6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421\": container with ID starting with 6d990e5f3e78b5e2dee611f50ecf1019c52431dca0ef47446ed7f187d11c5421 not found: ID does not exist" Sep 11 18:28:22.841849 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.841827 2579 scope.go:117] "RemoveContainer" containerID="41b47d8ed3b69323b9643d31e31b1915c735d627b1567b0361e9ef253ac0b65f" Sep 11 18:28:22.855995 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.855954 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw"] Sep 11 18:28:22.857868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.857841 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be394b-fbdc9b7c9-9d5gw"] Sep 11 18:28:22.869820 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.869785 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p"] Sep 11 18:28:22.874194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:22.874159 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-be394b-5b9dcb464d-9522p"] Sep 11 18:28:23.053596 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:23.053511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:23.761168 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:23.761136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:24.444383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:24.444346 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6488462f-742d-42d5-924f-05251050199b" path="/var/lib/kubelet/pods/6488462f-742d-42d5-924f-05251050199b/volumes" Sep 11 18:28:24.444790 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:24.444758 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ebddde-95e3-4462-aa05-86a3532aedf6" path="/var/lib/kubelet/pods/c8ebddde-95e3-4462-aa05-86a3532aedf6/volumes" Sep 11 18:28:24.566170 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:24.566133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:28:27.364563 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:28:27.364522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:04.336631 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:04.336601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:04.669693 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:04.669660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:10.546800 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:10.546766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:16.462158 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.462123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:16.530651 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.530617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:16.721493 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.721388 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj"] Sep 11 18:29:16.721812 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.721754 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" containerID="cri-o://141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315" gracePeriod=30 Sep 11 18:29:16.744734 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.744709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:16.861466 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.861409 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:29:16.861630 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:16.861493 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:29:17.277166 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:17.277139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:18.164072 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:18.164029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:19.049514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:19.049478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:19.775754 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:19.775721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:20.458138 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:20.458111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:21.966692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:21.966666 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:29:22.089844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.089762 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-token\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.089844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.089816 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istiod-ca-cert\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.089844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.089835 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-envoy\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.089874 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-podinfo\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.089910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-credential-socket\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090028 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-data\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090136 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl5qp\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-kube-api-access-kl5qp\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-certs\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-socket\") pod \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\" (UID: \"8c3f7042-cc4c-46e2-a7c3-a1e251205907\") " Sep 11 18:29:22.090409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090203 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:22.090409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:22.090409 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090307 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:29:22.090682 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090446 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:22.090682 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:22.090757 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090743 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.090804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090764 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.090804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090779 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.090804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090791 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.090906 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.090806 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.092085 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.092058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:29:22.092208 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.092125 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:29:22.092613 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.092594 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:29:22.092702 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.092623 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-kube-api-access-kl5qp" (OuterVolumeSpecName: "kube-api-access-kl5qp") pod "8c3f7042-cc4c-46e2-a7c3-a1e251205907" (UID: "8c3f7042-cc4c-46e2-a7c3-a1e251205907"). InnerVolumeSpecName "kube-api-access-kl5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:29:22.130842 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.130813 2579 generic.go:356] "Generic (PLEG): container finished" podID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerID="141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315" exitCode=0 Sep 11 18:29:22.130985 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.130861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" event={"ID":"8c3f7042-cc4c-46e2-a7c3-a1e251205907","Type":"ContainerDied","Data":"141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315"} Sep 11 18:29:22.130985 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.130874 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" Sep 11 18:29:22.130985 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.130931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj" event={"ID":"8c3f7042-cc4c-46e2-a7c3-a1e251205907","Type":"ContainerDied","Data":"9637ee5baeff2818b04ab1052937d78b5236845ae226679d8d122d47b8fa46dc"} Sep 11 18:29:22.130985 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.130953 2579 scope.go:117] "RemoveContainer" containerID="141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315" Sep 11 18:29:22.140174 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.140156 2579 scope.go:117] "RemoveContainer" containerID="0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466" Sep 11 18:29:22.148186 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.148170 2579 scope.go:117] "RemoveContainer" containerID="141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315" Sep 11 18:29:22.148462 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:29:22.148442 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315\": container with ID starting with 141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315 not found: ID does not exist" containerID="141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315" Sep 11 18:29:22.148513 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.148470 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315"} err="failed to get container status \"141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315\": rpc error: code = NotFound desc = could not find container \"141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315\": container with ID starting with 141e152919837e5f58463acb53913cc96549e7ddbb41d0bda40c9dc483f9d315 not found: ID does not exist" Sep 11 18:29:22.148513 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.148486 2579 scope.go:117] "RemoveContainer" containerID="0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466" Sep 11 18:29:22.148768 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:29:22.148749 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466\": container with ID starting with 0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466 not found: ID does not exist" containerID="0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466" Sep 11 18:29:22.148825 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.148772 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466"} err="failed to get container status \"0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466\": rpc error: code = NotFound desc = could not find container \"0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466\": container with ID starting with 0b875d6da7bdae74daba43063ce5e54609ca4d612b610607e550dc8e51139466 not found: ID does not exist" Sep 11 18:29:22.153280 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.153259 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj"] Sep 11 18:29:22.158464 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.158443 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-be394b-openshift-default-7d5b5c6fb-j4kzj"] Sep 11 18:29:22.192153 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.192132 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.192153 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.192153 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl5qp\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-kube-api-access-kl5qp\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.192288 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.192163 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.192288 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.192171 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c3f7042-cc4c-46e2-a7c3-a1e251205907-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:29:22.219244 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.219226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:22.443549 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:22.443509 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" path="/var/lib/kubelet/pods/8c3f7042-cc4c-46e2-a7c3-a1e251205907/volumes" Sep 11 18:29:27.745022 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:27.744992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:27.807256 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:27.807212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:27.908343 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:27.908312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:27.969255 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:27.969229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:28.354331 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:28.354295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:28.453551 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:28.453524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:29.082141 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:29.082101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:29.952436 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:29.952388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:34.521776 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:34.521738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:34.580552 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:34.580520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:34.800732 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:34.800655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:35.260098 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:35.260067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:36.381452 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:36.381405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:37.300945 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:37.300902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:38.058397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:38.058367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:43.757272 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:43.757236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:45.450093 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:45.450057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:46.758715 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:46.758682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:47.345458 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:47.345401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:53.014684 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:53.014652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:53.116751 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:53.116715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:53.466745 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:53.466714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:53.965167 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:53.965135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:54.691876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:54.691840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:29:55.562370 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:29:55.562329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.251094 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.251053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.282663 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.282630 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c"] Sep 11 18:30:04.282930 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.282896 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" podUID="71760337-fcd1-4cf7-bb83-f0b3992a467a" containerName="payload-processing" containerID="cri-o://00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b" gracePeriod=30 Sep 11 18:30:04.318105 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.318065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.383244 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.383207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.487476 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.487389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.548699 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.548669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.550038 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.550015 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:30:04.608266 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.608223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71760337-fcd1-4cf7-bb83-f0b3992a467a-plugins-config-volume\") pod \"71760337-fcd1-4cf7-bb83-f0b3992a467a\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " Sep 11 18:30:04.608518 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.608444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht2b8\" (UniqueName: \"kubernetes.io/projected/71760337-fcd1-4cf7-bb83-f0b3992a467a-kube-api-access-ht2b8\") pod \"71760337-fcd1-4cf7-bb83-f0b3992a467a\" (UID: \"71760337-fcd1-4cf7-bb83-f0b3992a467a\") " Sep 11 18:30:04.608736 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.608696 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71760337-fcd1-4cf7-bb83-f0b3992a467a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71760337-fcd1-4cf7-bb83-f0b3992a467a" (UID: "71760337-fcd1-4cf7-bb83-f0b3992a467a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:30:04.610896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.610872 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71760337-fcd1-4cf7-bb83-f0b3992a467a-kube-api-access-ht2b8" (OuterVolumeSpecName: "kube-api-access-ht2b8") pod "71760337-fcd1-4cf7-bb83-f0b3992a467a" (UID: "71760337-fcd1-4cf7-bb83-f0b3992a467a"). InnerVolumeSpecName "kube-api-access-ht2b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:30:04.709958 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.709926 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2b8\" (UniqueName: \"kubernetes.io/projected/71760337-fcd1-4cf7-bb83-f0b3992a467a-kube-api-access-ht2b8\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:04.709958 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.709956 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71760337-fcd1-4cf7-bb83-f0b3992a467a-plugins-config-volume\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:04.876628 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.876598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:04.976940 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:04.976906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:05.336567 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.336477 2579 generic.go:356] "Generic (PLEG): container finished" podID="71760337-fcd1-4cf7-bb83-f0b3992a467a" containerID="00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b" exitCode=0 Sep 11 18:30:05.336567 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.336521 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" Sep 11 18:30:05.337051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.336563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" event={"ID":"71760337-fcd1-4cf7-bb83-f0b3992a467a","Type":"ContainerDied","Data":"00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b"} Sep 11 18:30:05.337051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.336615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c" event={"ID":"71760337-fcd1-4cf7-bb83-f0b3992a467a","Type":"ContainerDied","Data":"dae03677158c0c18f7102f513028c7bf923bc10e95ee83b5bd89f058279dd29c"} Sep 11 18:30:05.337051 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.336640 2579 scope.go:117] "RemoveContainer" containerID="00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b" Sep 11 18:30:05.346856 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.346831 2579 scope.go:117] "RemoveContainer" containerID="00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b" Sep 11 18:30:05.347248 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:30:05.347219 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b\": container with ID starting with 00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b not found: ID does not exist" containerID="00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b" Sep 11 18:30:05.347368 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.347258 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b"} err="failed to get container status \"00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b\": rpc error: code = NotFound desc = could not find container \"00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b\": container with ID starting with 00cb02fa15b5c847d6409c2ec50ee979feb67de721c84401f6399d25e9134a6b not found: ID does not exist" Sep 11 18:30:05.362876 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.362845 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c"] Sep 11 18:30:05.370321 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:05.370295 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5d35e7-554f8fddbc-cr78c"] Sep 11 18:30:06.444068 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:06.444035 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71760337-fcd1-4cf7-bb83-f0b3992a467a" path="/var/lib/kubelet/pods/71760337-fcd1-4cf7-bb83-f0b3992a467a/volumes" Sep 11 18:30:11.207236 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.207148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:11.267495 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.267454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:11.453788 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.453758 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9"] Sep 11 18:30:11.454023 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.453998 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" containerID="cri-o://05825e3cf3c1e50a04b5af0fc374410b8707761ce2f8b9dc2a3bf6c73ae483cf" gracePeriod=30 Sep 11 18:30:11.483503 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.483402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:11.591778 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.591740 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 18:30:11.591941 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:11.591815 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 18:30:14.015078 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.015037 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-nvx9v"] Sep 11 18:30:14.015532 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.015240 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" podUID="923d603b-83c2-402f-acb6-4a85c45117a3" containerName="manager" containerID="cri-o://9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0" gracePeriod=30 Sep 11 18:30:14.057261 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.057231 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" podUID="923d603b-83c2-402f-acb6-4a85c45117a3" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.24:8081/readyz\": dial tcp 10.134.0.24:8081: connect: connection refused" Sep 11 18:30:14.261947 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.261924 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:30:14.381503 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.381473 2579 generic.go:356] "Generic (PLEG): container finished" podID="923d603b-83c2-402f-acb6-4a85c45117a3" containerID="9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0" exitCode=0 Sep 11 18:30:14.381696 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.381517 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" Sep 11 18:30:14.381696 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.381561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" event={"ID":"923d603b-83c2-402f-acb6-4a85c45117a3","Type":"ContainerDied","Data":"9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0"} Sep 11 18:30:14.381696 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.381610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-nvx9v" event={"ID":"923d603b-83c2-402f-acb6-4a85c45117a3","Type":"ContainerDied","Data":"46c31eaab3e49e558e9be2dd79c2b389855428ff9aa7471a9cd30188a3135b06"} Sep 11 18:30:14.381696 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.381642 2579 scope.go:117] "RemoveContainer" containerID="9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0" Sep 11 18:30:14.390973 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.390952 2579 scope.go:117] "RemoveContainer" containerID="9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0" Sep 11 18:30:14.391282 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:30:14.391258 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0\": container with ID starting with 9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0 not found: ID does not exist" containerID="9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0" Sep 11 18:30:14.391383 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.391292 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0"} err="failed to get container status \"9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0\": rpc error: code = NotFound desc = could not find container \"9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0\": container with ID starting with 9736d20684db5e84a411a2f8ebc341b63d98446908baa4931ba78675bd066ce0 not found: ID does not exist" Sep 11 18:30:14.423668 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.423634 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-metrics-certs\") pod \"923d603b-83c2-402f-acb6-4a85c45117a3\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " Sep 11 18:30:14.423822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.423703 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9klrf\" (UniqueName: \"kubernetes.io/projected/923d603b-83c2-402f-acb6-4a85c45117a3-kube-api-access-9klrf\") pod \"923d603b-83c2-402f-acb6-4a85c45117a3\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " Sep 11 18:30:14.423822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.423808 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-webhook-cert\") pod \"923d603b-83c2-402f-acb6-4a85c45117a3\" (UID: \"923d603b-83c2-402f-acb6-4a85c45117a3\") " Sep 11 18:30:14.425896 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.425863 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "923d603b-83c2-402f-acb6-4a85c45117a3" (UID: "923d603b-83c2-402f-acb6-4a85c45117a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:30:14.426009 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.425911 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923d603b-83c2-402f-acb6-4a85c45117a3-kube-api-access-9klrf" (OuterVolumeSpecName: "kube-api-access-9klrf") pod "923d603b-83c2-402f-acb6-4a85c45117a3" (UID: "923d603b-83c2-402f-acb6-4a85c45117a3"). InnerVolumeSpecName "kube-api-access-9klrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:30:14.426009 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.425938 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "923d603b-83c2-402f-acb6-4a85c45117a3" (UID: "923d603b-83c2-402f-acb6-4a85c45117a3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:30:14.525528 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.525489 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:14.525528 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.525531 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9klrf\" (UniqueName: \"kubernetes.io/projected/923d603b-83c2-402f-acb6-4a85c45117a3-kube-api-access-9klrf\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:14.525954 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.525547 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/923d603b-83c2-402f-acb6-4a85c45117a3-webhook-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:14.699157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.699066 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-nvx9v"] Sep 11 18:30:14.703117 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:14.703091 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-nvx9v"] Sep 11 18:30:16.444179 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.444143 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="923d603b-83c2-402f-acb6-4a85c45117a3" path="/var/lib/kubelet/pods/923d603b-83c2-402f-acb6-4a85c45117a3/volumes" Sep 11 18:30:16.702902 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.702846 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:30:16.845523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-credential-socket\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845523 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845529 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-socket\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845567 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-istio-token\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845599 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce032c31-6548-4e93-ab60-1463b3b02095-istio-podinfo\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845625 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-envoy\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce032c31-6548-4e93-ab60-1463b3b02095-istiod-ca-cert\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845748 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845745 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-data\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845774 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqd2m\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-kube-api-access-xqd2m\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845777 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:16.845994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-certs\") pod \"ce032c31-6548-4e93-ab60-1463b3b02095\" (UID: \"ce032c31-6548-4e93-ab60-1463b3b02095\") " Sep 11 18:30:16.845994 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.845908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:16.846221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.846113 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:16.846221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.846153 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-credential-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.846221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.846173 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-socket\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.846570 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.846277 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce032c31-6548-4e93-ab60-1463b3b02095-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:30:16.846718 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.846694 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:16.848069 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.848036 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce032c31-6548-4e93-ab60-1463b3b02095-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 18:30:16.848069 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.848064 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 18:30:16.848220 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.848084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:30:16.848220 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.848123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-kube-api-access-xqd2m" (OuterVolumeSpecName: "kube-api-access-xqd2m") pod "ce032c31-6548-4e93-ab60-1463b3b02095" (UID: "ce032c31-6548-4e93-ab60-1463b3b02095"). InnerVolumeSpecName "kube-api-access-xqd2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:30:16.946998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.946960 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce032c31-6548-4e93-ab60-1463b3b02095-istiod-ca-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.946998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.946990 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-data\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.946998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.946999 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqd2m\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-kube-api-access-xqd2m\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.946998 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.947008 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-workload-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.947277 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.947017 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce032c31-6548-4e93-ab60-1463b3b02095-istio-token\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.947277 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.947025 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce032c31-6548-4e93-ab60-1463b3b02095-istio-podinfo\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:16.947277 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:16.947033 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce032c31-6548-4e93-ab60-1463b3b02095-istio-envoy\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:30:17.279903 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.279870 2579 scope.go:117] "RemoveContainer" containerID="42b0a7d0d2d5e673568f9ab234ac315fcff20acda9e6fca261a9869cebf05f8e" Sep 11 18:30:17.288587 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.288567 2579 scope.go:117] "RemoveContainer" containerID="05825e3cf3c1e50a04b5af0fc374410b8707761ce2f8b9dc2a3bf6c73ae483cf" Sep 11 18:30:17.396692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.396656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" Sep 11 18:30:17.396692 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.396668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" event={"ID":"ce032c31-6548-4e93-ab60-1463b3b02095","Type":"ContainerDied","Data":"05825e3cf3c1e50a04b5af0fc374410b8707761ce2f8b9dc2a3bf6c73ae483cf"} Sep 11 18:30:17.396943 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.396725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9" event={"ID":"ce032c31-6548-4e93-ab60-1463b3b02095","Type":"ContainerDied","Data":"1e0817bd411eda6510bb3e6d06492d67bb3bc56c068314eb59579f249aae426e"} Sep 11 18:30:17.434140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.434103 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9"] Sep 11 18:30:17.438106 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:17.438076 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5d35e7-openshift-default-5549f7c85b-hbrb9"] Sep 11 18:30:18.442929 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:18.442895 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" path="/var/lib/kubelet/pods/ce032c31-6548-4e93-ab60-1463b3b02095/volumes" Sep 11 18:30:59.475397 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:59.475365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:30:59.537676 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:30:59.537645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:02.815799 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:02.815761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:15.522379 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522347 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9b9f55b89-wvxr2"] Sep 11 18:31:15.522915 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522864 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" Sep 11 18:31:15.522915 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522884 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" Sep 11 18:31:15.522915 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522896 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" Sep 11 18:31:15.522915 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522904 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522919 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6488462f-742d-42d5-924f-05251050199b" containerName="maas-api" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522928 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6488462f-742d-42d5-924f-05251050199b" containerName="maas-api" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522941 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="923d603b-83c2-402f-acb6-4a85c45117a3" containerName="manager" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522950 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="923d603b-83c2-402f-acb6-4a85c45117a3" containerName="manager" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522962 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522970 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522981 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.522990 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523001 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71760337-fcd1-4cf7-bb83-f0b3992a467a" containerName="payload-processing" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523009 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71760337-fcd1-4cf7-bb83-f0b3992a467a" containerName="payload-processing" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523025 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ebddde-95e3-4462-aa05-86a3532aedf6" containerName="payload-processing" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523033 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ebddde-95e3-4462-aa05-86a3532aedf6" containerName="payload-processing" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523050 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" Sep 11 18:31:15.523140 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523059 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523162 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523175 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ebddde-95e3-4462-aa05-86a3532aedf6" containerName="payload-processing" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523186 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523194 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523202 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71760337-fcd1-4cf7-bb83-f0b3992a467a" containerName="payload-processing" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523212 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6488462f-742d-42d5-924f-05251050199b" containerName="maas-api" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523220 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3f7042-cc4c-46e2-a7c3-a1e251205907" containerName="istio-proxy" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523229 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="923d603b-83c2-402f-acb6-4a85c45117a3" containerName="manager" Sep 11 18:31:15.523844 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.523237 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" Sep 11 18:31:15.526557 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.526529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.531435 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.531395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 18:31:15.531660 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.531639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 18:31:15.531660 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.531658 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vtvpv\"" Sep 11 18:31:15.531813 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.531667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 18:31:15.537761 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.537732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-wvxr2"] Sep 11 18:31:15.579906 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.579877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72v4x\" (UniqueName: \"kubernetes.io/projected/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-kube-api-access-72v4x\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.580063 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.579927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-webhook-cert\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.580063 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.580020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-metrics-certs\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.681247 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.681210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72v4x\" (UniqueName: \"kubernetes.io/projected/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-kube-api-access-72v4x\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.681487 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.681262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-webhook-cert\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.681487 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.681299 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-metrics-certs\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.683953 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.683922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-metrics-certs\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.684061 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.683955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-webhook-cert\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.691355 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.691333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72v4x\" (UniqueName: \"kubernetes.io/projected/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-kube-api-access-72v4x\") pod \"maas-controller-9b9f55b89-wvxr2\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:15.837749 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:15.837660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:16.118819 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.118788 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-wvxr2"] Sep 11 18:31:16.123278 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.123255 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 18:31:16.533764 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.533742 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 18:31:16.654889 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.654853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" event={"ID":"6bd7ef1c-788e-4d26-af5c-52233cd9bc10","Type":"ContainerStarted","Data":"9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb"} Sep 11 18:31:16.655067 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.654905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" event={"ID":"6bd7ef1c-788e-4d26-af5c-52233cd9bc10","Type":"ContainerStarted","Data":"1a93e82f858305b23ffc1437ab0d8b85eec664ebd1a3b9a29bbae8848a401105"} Sep 11 18:31:16.655067 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.654954 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:16.678058 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:16.678001 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" podStartSLOduration=1.270708919 podStartE2EDuration="1.677985705s" totalImagesPullingTime="407.276786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:16.676324297 +0000 UTC m=+2220.707385551" watchObservedRunningTime="2026-09-11 18:31:16.677985705 +0000 UTC m=+2220.709046958" Sep 11 18:31:27.661703 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:27.661672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:39.023499 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.023384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:39.124485 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.124456 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-wvxr2"] Sep 11 18:31:39.124723 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.124700 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" podUID="6bd7ef1c-788e-4d26-af5c-52233cd9bc10" containerName="manager" containerID="cri-o://9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb" gracePeriod=30 Sep 11 18:31:39.369328 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.369307 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:39.503654 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.503616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72v4x\" (UniqueName: \"kubernetes.io/projected/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-kube-api-access-72v4x\") pod \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " Sep 11 18:31:39.503842 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.503748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-webhook-cert\") pod \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " Sep 11 18:31:39.503842 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.503791 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-metrics-certs\") pod \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\" (UID: \"6bd7ef1c-788e-4d26-af5c-52233cd9bc10\") " Sep 11 18:31:39.505809 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.505780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-kube-api-access-72v4x" (OuterVolumeSpecName: "kube-api-access-72v4x") pod "6bd7ef1c-788e-4d26-af5c-52233cd9bc10" (UID: "6bd7ef1c-788e-4d26-af5c-52233cd9bc10"). InnerVolumeSpecName "kube-api-access-72v4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:31:39.505937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.505827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6bd7ef1c-788e-4d26-af5c-52233cd9bc10" (UID: "6bd7ef1c-788e-4d26-af5c-52233cd9bc10"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:31:39.505937 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.505868 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6bd7ef1c-788e-4d26-af5c-52233cd9bc10" (UID: "6bd7ef1c-788e-4d26-af5c-52233cd9bc10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:31:39.605471 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.605356 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-webhook-cert\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:31:39.605471 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.605392 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:31:39.605471 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.605405 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72v4x\" (UniqueName: \"kubernetes.io/projected/6bd7ef1c-788e-4d26-af5c-52233cd9bc10-kube-api-access-72v4x\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:31:39.757221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.757184 2579 generic.go:356] "Generic (PLEG): container finished" podID="6bd7ef1c-788e-4d26-af5c-52233cd9bc10" containerID="9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb" exitCode=0 Sep 11 18:31:39.757456 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.757231 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" Sep 11 18:31:39.757456 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.757225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" event={"ID":"6bd7ef1c-788e-4d26-af5c-52233cd9bc10","Type":"ContainerDied","Data":"9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb"} Sep 11 18:31:39.757456 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.757294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-wvxr2" event={"ID":"6bd7ef1c-788e-4d26-af5c-52233cd9bc10","Type":"ContainerDied","Data":"1a93e82f858305b23ffc1437ab0d8b85eec664ebd1a3b9a29bbae8848a401105"} Sep 11 18:31:39.757456 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.757324 2579 scope.go:117] "RemoveContainer" containerID="9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb" Sep 11 18:31:39.766361 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.766343 2579 scope.go:117] "RemoveContainer" containerID="9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb" Sep 11 18:31:39.766650 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:31:39.766631 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb\": container with ID starting with 9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb not found: ID does not exist" containerID="9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb" Sep 11 18:31:39.766709 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.766659 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb"} err="failed to get container status \"9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb\": rpc error: code = NotFound desc = could not find container \"9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb\": container with ID starting with 9667c87dd70df3871ea6d3cfe733a87b3be3081d61f3bc3a5f89919f962447fb not found: ID does not exist" Sep 11 18:31:39.780707 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.780684 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-wvxr2"] Sep 11 18:31:39.784666 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:39.784641 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-wvxr2"] Sep 11 18:31:40.440454 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.440403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:31:40.443293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.443270 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd7ef1c-788e-4d26-af5c-52233cd9bc10" path="/var/lib/kubelet/pods/6bd7ef1c-788e-4d26-af5c-52233cd9bc10/volumes" Sep 11 18:31:40.562828 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.562793 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9b9f55b89-vmcr8"] Sep 11 18:31:40.563147 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.563135 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" Sep 11 18:31:40.563196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.563148 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce032c31-6548-4e93-ab60-1463b3b02095" containerName="istio-proxy" Sep 11 18:31:40.563196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.563163 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd7ef1c-788e-4d26-af5c-52233cd9bc10" containerName="manager" Sep 11 18:31:40.563196 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.563169 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd7ef1c-788e-4d26-af5c-52233cd9bc10" containerName="manager" Sep 11 18:31:40.563290 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.563227 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac59e7ad-65d1-454c-8130-b2b0443ea379" containerName="istio-proxy" Sep 11 18:31:40.563290 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.563234 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd7ef1c-788e-4d26-af5c-52233cd9bc10" containerName="manager" Sep 11 18:31:40.567514 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.567494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.572986 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.572939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 11 18:31:40.573103 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.573001 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 11 18:31:40.573163 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.573153 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vtvpv\"" Sep 11 18:31:40.573200 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.573006 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 11 18:31:40.579155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.579134 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-vmcr8"] Sep 11 18:31:40.614526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.614501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3319f82-eaad-4685-b15c-f8d946bbe654-webhook-cert\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.614672 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.614553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3319f82-eaad-4685-b15c-f8d946bbe654-metrics-certs\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.614672 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.614574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkv7\" (UniqueName: \"kubernetes.io/projected/a3319f82-eaad-4685-b15c-f8d946bbe654-kube-api-access-cfkv7\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.715500 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.715385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3319f82-eaad-4685-b15c-f8d946bbe654-metrics-certs\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.715500 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.715472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkv7\" (UniqueName: \"kubernetes.io/projected/a3319f82-eaad-4685-b15c-f8d946bbe654-kube-api-access-cfkv7\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.715752 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.715573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3319f82-eaad-4685-b15c-f8d946bbe654-webhook-cert\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.718076 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.718050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3319f82-eaad-4685-b15c-f8d946bbe654-webhook-cert\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.718169 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.718084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3319f82-eaad-4685-b15c-f8d946bbe654-metrics-certs\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.724072 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.724048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkv7\" (UniqueName: \"kubernetes.io/projected/a3319f82-eaad-4685-b15c-f8d946bbe654-kube-api-access-cfkv7\") pod \"maas-controller-9b9f55b89-vmcr8\" (UID: \"a3319f82-eaad-4685-b15c-f8d946bbe654\") " pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:40.879287 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:40.879254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:41.155968 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:41.155935 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9b9f55b89-vmcr8"] Sep 11 18:31:41.157222 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:31:41.157198 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3319f82_eaad_4685_b15c_f8d946bbe654.slice/crio-f46d3ff2802c689b93cb343398e7d48b6a5aaf14fb0fbae4ae55373700bd591c WatchSource:0}: Error finding container f46d3ff2802c689b93cb343398e7d48b6a5aaf14fb0fbae4ae55373700bd591c: Status 404 returned error can't find the container with id f46d3ff2802c689b93cb343398e7d48b6a5aaf14fb0fbae4ae55373700bd591c Sep 11 18:31:41.768734 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:41.768702 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" event={"ID":"a3319f82-eaad-4685-b15c-f8d946bbe654","Type":"ContainerStarted","Data":"42ab4ff4437d5696c444700a830585a9ac691391800b2eba58f1734db8a2be3d"} Sep 11 18:31:41.769112 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:41.768753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" event={"ID":"a3319f82-eaad-4685-b15c-f8d946bbe654","Type":"ContainerStarted","Data":"f46d3ff2802c689b93cb343398e7d48b6a5aaf14fb0fbae4ae55373700bd591c"} Sep 11 18:31:41.769112 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:41.768781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:31:41.791157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:41.791087 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" podStartSLOduration=1.400881649 podStartE2EDuration="1.791066033s" totalImagesPullingTime="390.184384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:41.790830442 +0000 UTC m=+2245.821891688" watchObservedRunningTime="2026-09-11 18:31:41.791066033 +0000 UTC m=+2245.822127266" Sep 11 18:31:52.775881 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:31:52.775851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9b9f55b89-vmcr8" Sep 11 18:32:19.529293 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:32:19.529254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:32:31.073747 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:32:31.073699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:49.658440 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:33:49.658385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:33:49.731026 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:33:49.730993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:19.368715 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:34:19.368679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:29.246228 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:34:29.246195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:34.747274 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:34:34.747233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:49.131347 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:34:49.131257 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:34:50.645298 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:34:50.645260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:07.162586 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.162551 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b"] Sep 11 18:35:07.166178 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.166152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.169552 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.169526 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2sf5b\"" Sep 11 18:35:07.185378 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.185341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b"] Sep 11 18:35:07.263858 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.263821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-metrics-certs\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.264045 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.263864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-openshift-service-ca\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.264045 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.263917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwx7g\" (UniqueName: \"kubernetes.io/projected/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-kube-api-access-lwx7g\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.264045 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.263991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-maas-api-tls\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.365540 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.365493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-maas-api-tls\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.365768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.365587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-metrics-certs\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.365768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.365637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-openshift-service-ca\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.365768 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.365679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwx7g\" (UniqueName: \"kubernetes.io/projected/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-kube-api-access-lwx7g\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.366502 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.366322 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-openshift-service-ca\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.368209 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.368178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-maas-api-tls\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.368319 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.368237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-metrics-certs\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.376444 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.376393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwx7g\" (UniqueName: \"kubernetes.io/projected/9e1b1a7c-1ac6-4d48-a1ee-959109c01494-kube-api-access-lwx7g\") pod \"maas-api-bc5d8db47-b4h8b\" (UID: \"9e1b1a7c-1ac6-4d48-a1ee-959109c01494\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.478064 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.477975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:07.751952 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:07.751928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b"] Sep 11 18:35:07.753832 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:35:07.753807 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1b1a7c_1ac6_4d48_a1ee_959109c01494.slice/crio-2119e3cbd96136195e7a7bb60754b87008744795dc8311d827aa30cf7e50fcce WatchSource:0}: Error finding container 2119e3cbd96136195e7a7bb60754b87008744795dc8311d827aa30cf7e50fcce: Status 404 returned error can't find the container with id 2119e3cbd96136195e7a7bb60754b87008744795dc8311d827aa30cf7e50fcce Sep 11 18:35:08.704198 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:08.704150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" event={"ID":"9e1b1a7c-1ac6-4d48-a1ee-959109c01494","Type":"ContainerStarted","Data":"8fd6ba8cd4a43f546989a20d21c0421874cfb6b45760bb454f4592d6da9081ad"} Sep 11 18:35:08.704600 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:08.704214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" event={"ID":"9e1b1a7c-1ac6-4d48-a1ee-959109c01494","Type":"ContainerStarted","Data":"2119e3cbd96136195e7a7bb60754b87008744795dc8311d827aa30cf7e50fcce"} Sep 11 18:35:08.704600 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:08.704249 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:08.724728 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:08.724674 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" podStartSLOduration=1.229232888 podStartE2EDuration="1.724657469s" totalImagesPullingTime="495.424581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:08.722802038 +0000 UTC m=+2452.753863323" watchObservedRunningTime="2026-09-11 18:35:08.724657469 +0000 UTC m=+2452.755718699" Sep 11 18:35:14.713583 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:14.713555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-b4h8b" Sep 11 18:35:22.447881 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:22.447848 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw"] Sep 11 18:35:22.448325 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:22.448030 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" podUID="224e23ce-68eb-4eb9-8cff-e799380085df" containerName="maas-api" containerID="cri-o://407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4" gracePeriod=30 Sep 11 18:35:23.215596 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.215557 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" podUID="224e23ce-68eb-4eb9-8cff-e799380085df" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.25:8443/health\": dial tcp 10.134.0.25:8443: connect: connection refused" Sep 11 18:35:23.694076 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.694052 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:35:23.778606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.778526 2579 generic.go:356] "Generic (PLEG): container finished" podID="224e23ce-68eb-4eb9-8cff-e799380085df" containerID="407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4" exitCode=0 Sep 11 18:35:23.778606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.778575 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" Sep 11 18:35:23.778606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.778586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" event={"ID":"224e23ce-68eb-4eb9-8cff-e799380085df","Type":"ContainerDied","Data":"407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4"} Sep 11 18:35:23.778824 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.778627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw" event={"ID":"224e23ce-68eb-4eb9-8cff-e799380085df","Type":"ContainerDied","Data":"522177bc3041ca2b068763f84d2a0734915b2b99ebf2d71643fe04f892620b19"} Sep 11 18:35:23.778824 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.778649 2579 scope.go:117] "RemoveContainer" containerID="407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4" Sep 11 18:35:23.787128 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.787107 2579 scope.go:117] "RemoveContainer" containerID="407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4" Sep 11 18:35:23.787391 ip-10-0-141-255 kubenswrapper[2579]: E0911 18:35:23.787373 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4\": container with ID starting with 407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4 not found: ID does not exist" containerID="407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4" Sep 11 18:35:23.787506 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.787401 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4"} err="failed to get container status \"407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4\": rpc error: code = NotFound desc = could not find container \"407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4\": container with ID starting with 407000d312634a30867a37c9f454c0b2b30c8dea9ac674ace3724ec72586fff4 not found: ID does not exist" Sep 11 18:35:23.814797 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.814772 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgkgh\" (UniqueName: \"kubernetes.io/projected/224e23ce-68eb-4eb9-8cff-e799380085df-kube-api-access-kgkgh\") pod \"224e23ce-68eb-4eb9-8cff-e799380085df\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " Sep 11 18:35:23.814905 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.814862 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224e23ce-68eb-4eb9-8cff-e799380085df-openshift-service-ca\") pod \"224e23ce-68eb-4eb9-8cff-e799380085df\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " Sep 11 18:35:23.814974 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.814959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-maas-api-tls\") pod \"224e23ce-68eb-4eb9-8cff-e799380085df\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " Sep 11 18:35:23.815032 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.815010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-metrics-certs\") pod \"224e23ce-68eb-4eb9-8cff-e799380085df\" (UID: \"224e23ce-68eb-4eb9-8cff-e799380085df\") " Sep 11 18:35:23.815180 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.815161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224e23ce-68eb-4eb9-8cff-e799380085df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "224e23ce-68eb-4eb9-8cff-e799380085df" (UID: "224e23ce-68eb-4eb9-8cff-e799380085df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 18:35:23.815279 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.815265 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224e23ce-68eb-4eb9-8cff-e799380085df-openshift-service-ca\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:35:23.817077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.817048 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "224e23ce-68eb-4eb9-8cff-e799380085df" (UID: "224e23ce-68eb-4eb9-8cff-e799380085df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:35:23.817077 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.817071 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "224e23ce-68eb-4eb9-8cff-e799380085df" (UID: "224e23ce-68eb-4eb9-8cff-e799380085df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 18:35:23.817221 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.817142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224e23ce-68eb-4eb9-8cff-e799380085df-kube-api-access-kgkgh" (OuterVolumeSpecName: "kube-api-access-kgkgh") pod "224e23ce-68eb-4eb9-8cff-e799380085df" (UID: "224e23ce-68eb-4eb9-8cff-e799380085df"). InnerVolumeSpecName "kube-api-access-kgkgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 18:35:23.916346 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.916312 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-maas-api-tls\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:35:23.916346 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.916336 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224e23ce-68eb-4eb9-8cff-e799380085df-metrics-certs\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:35:23.916346 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:23.916346 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgkgh\" (UniqueName: \"kubernetes.io/projected/224e23ce-68eb-4eb9-8cff-e799380085df-kube-api-access-kgkgh\") on node \"ip-10-0-141-255.ec2.internal\" DevicePath \"\"" Sep 11 18:35:24.102498 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:24.102461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw"] Sep 11 18:35:24.106501 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:24.106477 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-q8qmw"] Sep 11 18:35:24.444253 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:24.444212 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224e23ce-68eb-4eb9-8cff-e799380085df" path="/var/lib/kubelet/pods/224e23ce-68eb-4eb9-8cff-e799380085df/volumes" Sep 11 18:35:42.433709 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:42.433681 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9b9f55b89-vmcr8_a3319f82-eaad-4685-b15c-f8d946bbe654/manager/0.log" Sep 11 18:35:43.041801 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:43.041770 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v_47fd9dbe-66b1-4263-9dd6-1297bc874315/manager/0.log" Sep 11 18:35:43.061784 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:43.061764 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v_47fd9dbe-66b1-4263-9dd6-1297bc874315/copy-manifests/0.log" Sep 11 18:35:43.386531 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:43.386508 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bc5d8db47-b4h8b_9e1b1a7c-1ac6-4d48-a1ee-959109c01494/maas-api/0.log" Sep 11 18:35:45.651953 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:45.651922 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvgj87_43ee41ff-9811-4d0c-b51d-968f357d8887/istio-proxy/0.log" Sep 11 18:35:45.776540 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:45.776514 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r7bxr_1990330b-607a-43a2-8990-fe64a6ae016e/discovery/0.log" Sep 11 18:35:46.230606 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:46.230573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dnhtk_45a694aa-f5b7-40dc-9c90-3869ab675962/payload-pre-processing/0.log" Sep 11 18:35:46.463504 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:46.463475 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64477965b8-nbp56_66738241-d01b-4e83-9c45-1be1afac1e1e/router/0.log" Sep 11 18:35:46.966568 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:46.966535 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nvwkw_74cd62a3-ddff-4fe7-bc66-d1165e22e3cc/main/0.log" Sep 11 18:35:47.359164 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:47.359082 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp29n8_b2ca90fc-31f5-4375-8c1b-155912f7ea56/main/0.log" Sep 11 18:35:52.815490 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:52.815455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:52.943573 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:52.943504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:56.006402 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:56.006365 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z99jr_e12fa326-e7dc-47bf-93d9-dd1f0344a2c5/global-pull-secret-syncer/0.log" Sep 11 18:35:56.062192 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:56.062160 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" Sep 11 18:35:56.205393 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:56.205359 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-255.ec2.internal_8b998e155b62d16440a868b586601783/haproxy/0.log" Sep 11 18:35:59.604454 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:59.604401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:35:59.698802 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:35:59.698768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gxrw_85d370f9-8788-4fde-af01-dad42dccac2c/konnectivity-agent/0.log" limit=1024 Sep 11 18:36:02.605575 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:02.605539 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-84p4m_fe9495a1-b461-401a-bb7e-8c177b56c9bf/kube-state-metrics/0.log" Sep 11 18:36:02.637103 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:02.637063 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-84p4m_fe9495a1-b461-401a-bb7e-8c177b56c9bf/kube-rbac-proxy-main/0.log" Sep 11 18:36:02.670613 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:02.670581 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-84p4m_fe9495a1-b461-401a-bb7e-8c177b56c9bf/kube-rbac-proxy-self/0.log" Sep 11 18:36:02.806375 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:02.806346 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ckmm_e33cb2f7-0e24-4b5e-b44a-d85c5e018e48/node-exporter/0.log" Sep 11 18:36:02.838930 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:02.838903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ckmm_e33cb2f7-0e24-4b5e-b44a-d85c5e018e48/kube-rbac-proxy/0.log" Sep 11 18:36:02.874662 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:02.874633 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ckmm_e33cb2f7-0e24-4b5e-b44a-d85c5e018e48/init-textfile/0.log" Sep 11 18:36:03.308904 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.308872 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/prometheus/0.log" Sep 11 18:36:03.341812 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.341796 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/config-reloader/0.log" Sep 11 18:36:03.377542 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.377516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/thanos-sidecar/0.log" Sep 11 18:36:03.407154 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.407130 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/kube-rbac-proxy-web/0.log" Sep 11 18:36:03.446121 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.446092 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/kube-rbac-proxy/0.log" Sep 11 18:36:03.493537 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.493509 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/kube-rbac-proxy-thanos/0.log" Sep 11 18:36:03.525577 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.525545 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cd7e049c-e58d-4b14-bf99-ca4f0148fc47/init-config-reloader/0.log" Sep 11 18:36:03.643654 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.643626 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-4xqbm_585b8669-f805-474d-9b75-d97d91055862/prometheus-operator-admission-webhook/0.log" Sep 11 18:36:03.831550 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.831520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8499f76744-l7s99_dd9d268b-fc51-4b1c-9214-beed628132a3/thanos-query/0.log" Sep 11 18:36:03.859804 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.859782 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8499f76744-l7s99_dd9d268b-fc51-4b1c-9214-beed628132a3/kube-rbac-proxy-web/0.log" Sep 11 18:36:03.921760 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.921677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8499f76744-l7s99_dd9d268b-fc51-4b1c-9214-beed628132a3/kube-rbac-proxy/0.log" Sep 11 18:36:03.982288 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:03.982258 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8499f76744-l7s99_dd9d268b-fc51-4b1c-9214-beed628132a3/prom-label-proxy/0.log" Sep 11 18:36:04.032395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:04.032367 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8499f76744-l7s99_dd9d268b-fc51-4b1c-9214-beed628132a3/kube-rbac-proxy-rules/0.log" Sep 11 18:36:04.089305 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:04.089281 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8499f76744-l7s99_dd9d268b-fc51-4b1c-9214-beed628132a3/kube-rbac-proxy-metrics/0.log" Sep 11 18:36:05.693520 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.693481 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp"] Sep 11 18:36:05.694005 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.693986 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224e23ce-68eb-4eb9-8cff-e799380085df" containerName="maas-api" Sep 11 18:36:05.694081 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.694007 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="224e23ce-68eb-4eb9-8cff-e799380085df" containerName="maas-api" Sep 11 18:36:05.694157 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.694141 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="224e23ce-68eb-4eb9-8cff-e799380085df" containerName="maas-api" Sep 11 18:36:05.697500 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.697476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.709806 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.709775 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt88g\"/\"kube-root-ca.crt\"" Sep 11 18:36:05.710049 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.710018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mt88g\"/\"default-dockercfg-2kcn4\"" Sep 11 18:36:05.710155 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.710042 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt88g\"/\"openshift-service-ca.crt\"" Sep 11 18:36:05.716632 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.716608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp"] Sep 11 18:36:05.803919 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.803882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-lib-modules\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.804120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.803937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-podres\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.804120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.804012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nxk\" (UniqueName: \"kubernetes.io/projected/ac97c153-6d09-4df0-bb5e-b155cb81a393-kube-api-access-b9nxk\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.804120 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.804089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-proc\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.804258 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.804141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-sys\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-proc\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905084 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-sys\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-lib-modules\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905173 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-proc\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-podres\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-sys\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nxk\" (UniqueName: \"kubernetes.io/projected/ac97c153-6d09-4df0-bb5e-b155cb81a393-kube-api-access-b9nxk\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905299 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-podres\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.905532 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.905294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac97c153-6d09-4df0-bb5e-b155cb81a393-lib-modules\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:05.918112 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:05.918087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nxk\" (UniqueName: \"kubernetes.io/projected/ac97c153-6d09-4df0-bb5e-b155cb81a393-kube-api-access-b9nxk\") pod \"perf-node-gather-daemonset-f5pgp\" (UID: \"ac97c153-6d09-4df0-bb5e-b155cb81a393\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:06.011838 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:06.011733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:06.287682 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:06.287653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp"] Sep 11 18:36:06.289870 ip-10-0-141-255 kubenswrapper[2579]: W0911 18:36:06.289837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac97c153_6d09_4df0_bb5e_b155cb81a393.slice/crio-c78d6f0d19cf9362f86cf174c623ba1f861b2fe6f40f5586030f2065b0420701 WatchSource:0}: Error finding container c78d6f0d19cf9362f86cf174c623ba1f861b2fe6f40f5586030f2065b0420701: Status 404 returned error can't find the container with id c78d6f0d19cf9362f86cf174c623ba1f861b2fe6f40f5586030f2065b0420701 Sep 11 18:36:06.971095 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:06.971053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" event={"ID":"ac97c153-6d09-4df0-bb5e-b155cb81a393","Type":"ContainerStarted","Data":"1dc49c9edf2d67e55c1ba98e7220ecfa58f087fb26869d76aab185e638bb4858"} Sep 11 18:36:06.971526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:06.971111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" event={"ID":"ac97c153-6d09-4df0-bb5e-b155cb81a393","Type":"ContainerStarted","Data":"c78d6f0d19cf9362f86cf174c623ba1f861b2fe6f40f5586030f2065b0420701"} Sep 11 18:36:06.971526 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:06.971215 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:07.000357 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:07.000306 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" podStartSLOduration=2.00029224 podStartE2EDuration="2.00029224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:06.998323541 +0000 UTC m=+2511.029384791" watchObservedRunningTime="2026-09-11 18:36:07.00029224 +0000 UTC m=+2511.031353504" Sep 11 18:36:08.782395 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:08.782325 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wlz59_522b462e-cb2f-4b6a-9a16-d84758954478/dns/0.log" Sep 11 18:36:08.806099 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:08.806059 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wlz59_522b462e-cb2f-4b6a-9a16-d84758954478/kube-rbac-proxy/0.log" Sep 11 18:36:08.880883 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:08.880850 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t5fnc_e3c2881f-9312-4856-9964-5d0c94db3ea8/dns-node-resolver/0.log" Sep 11 18:36:09.452366 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:09.452338 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55b6cdf8db-d84sb_253a9bc2-5bea-4a7f-8ff6-dd21410eb2a5/registry/0.log" Sep 11 18:36:09.527787 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:09.527753 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrnd9_93fceeb3-bb73-4ec4-956d-553a85b9f92a/node-ca/0.log" Sep 11 18:36:10.655594 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:10.655564 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvgj87_43ee41ff-9811-4d0c-b51d-968f357d8887/istio-proxy/0.log" Sep 11 18:36:11.107040 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:11.106952 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r7bxr_1990330b-607a-43a2-8990-fe64a6ae016e/discovery/0.log" Sep 11 18:36:11.239194 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:11.239162 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dnhtk_45a694aa-f5b7-40dc-9c90-3869ab675962/payload-pre-processing/0.log" Sep 11 18:36:11.307368 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:11.307339 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64477965b8-nbp56_66738241-d01b-4e83-9c45-1be1afac1e1e/router/0.log" Sep 11 18:36:12.221541 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:12.221516 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9pnbx_4b8b095b-591c-4f3d-97c6-bc7b5833ab38/serve-healthcheck-canary/0.log" Sep 11 18:36:12.891836 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:12.891805 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zn7n_c0cf0486-0899-4578-abb2-3667f19ad281/kube-rbac-proxy/0.log" Sep 11 18:36:12.941566 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:12.941537 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zn7n_c0cf0486-0899-4578-abb2-3667f19ad281/exporter/0.log" Sep 11 18:36:12.986256 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:12.986223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-f5pgp" Sep 11 18:36:13.020561 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:13.020537 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5zn7n_c0cf0486-0899-4578-abb2-3667f19ad281/extractor/0.log" Sep 11 18:36:15.960973 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:15.960944 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9b9f55b89-vmcr8_a3319f82-eaad-4685-b15c-f8d946bbe654/manager/0.log" Sep 11 18:36:16.289215 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:16.289182 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvgj87_43ee41ff-9811-4d0c-b51d-968f357d8887/istio-proxy/0.log" Sep 11 18:36:16.412747 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:16.412634 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v_47fd9dbe-66b1-4263-9dd6-1297bc874315/manager/0.log" Sep 11 18:36:16.433355 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:16.433326 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-xkn5v_47fd9dbe-66b1-4263-9dd6-1297bc874315/copy-manifests/0.log" Sep 11 18:36:16.750332 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:16.750300 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r7bxr_1990330b-607a-43a2-8990-fe64a6ae016e/discovery/0.log" Sep 11 18:36:16.855190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:16.855162 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dnhtk_45a694aa-f5b7-40dc-9c90-3869ab675962/payload-pre-processing/0.log" Sep 11 18:36:16.923483 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:16.923457 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64477965b8-nbp56_66738241-d01b-4e83-9c45-1be1afac1e1e/router/0.log" Sep 11 18:36:17.948060 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:17.948029 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ffbd766d4-jc9pj_b6329017-16da-4de4-aee0-14082b9f0a0f/manager/0.log" Sep 11 18:36:18.649582 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:18.649549 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvgj87_43ee41ff-9811-4d0c-b51d-968f357d8887/istio-proxy/0.log" Sep 11 18:36:19.094868 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:19.094787 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r7bxr_1990330b-607a-43a2-8990-fe64a6ae016e/discovery/0.log" Sep 11 18:36:19.206190 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:19.206162 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dnhtk_45a694aa-f5b7-40dc-9c90-3869ab675962/payload-pre-processing/0.log" Sep 11 18:36:19.265822 ip-10-0-141-255 kubenswrapper[2579]: I0911 18:36:19.265793 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64477965b8-nbp56_66738241-d01b-4e83-9c45-1be1afac1e1e/router/0.log"